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Abstract 

Balancing  task  allocation  between  humans  and  computers  is  crucial  to  the  development  of  effective 
decision  support  systems.  This  thesis  investigates  the  appropriate  balance  between  humans  and 
automation  for  geospatial  path  problem  solving  within  the  high-risk  domain  of  human  planetary 
surface  exploration,  where  decisions  are  time  critical  and  humans  must  adapt  to  uncertainty.  In 
order  to  develop  flexible  and  robust  decision  support  systems  for  Lunar  and  planetary  exploration, 
human-automation  role  allocations  are  examined  to  understand  how  humans  conduct  complex 
optimizations  under  different  degrees  of  automated  assistance. 

A  work  domain  analysis  of  human  planetary  extravehicular  activities  (EVA)  resulted  in  a  framework 
for  human-robotic  planning,  including  key  input  variables,  constraints,  and  outputs.  Based  on  this 
analysis,  a  prototype  path  planning  aid  was  developed.  Under  investigation  was  the  use  of  partial 
automatic  path  generation  and  a  visualization  called  levels  of  equal  cost  (LOEC,  an  aggregate  cost 
map).  Human-in-the-loop  testing  was  employed  to  understand  the  effects  of  the  automated 
assistance  and  different  visualizations  on  path  planning  performance  across  multivariate  cost 
functions.  In  two  separate  experiments,  participants  were  tasked  to  make  obstacle-free,  least-costly 
paths  based  on  given  cost  functions. 

Analysis  of  the  experimental  results  indicated  that  knowledge-based  reasoning  is  best  supported 
when  operators  conduct  manual  sensitivity  analysis,  a  strategy  that  was  absent  when  path  generation 
was  allocated  to  automation.  Leveraging  computer-generated  paths  resulted  in  overall  better  path 
performance  but  also  led  to  automation  bias  and  decreased  situation  awareness.  With  respect  to 
visualizations,  participants  using  elevation  contours  had  lower  cost  paths  and  short  task  times  when 
automation  was  reliable.  However,  the  LOEC  visualization  helped  participants  initially  create  least- 
costly  paths  for  the  most  complex  cost  function.  Furthermore,  LOEC  visualization  appears  to 
promote  manual  sensitivity  analysis,  which  was  beneficial  in  the  degraded  automation  condition, 
where  low  cost  and  time  was  observed  for  participants  with  this  visualization.  Finally,  two  types  of 
sensitivity  analysis  were  observed,  one  that  leveraged  the  available  “what  if’  tools  and  the  other  that 
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1  Introduction 


1.1  Motivation 

On  May  25,  1961,  President  John  F.  Kennedy  endeavored  the  United  States  to  send  a  man  to  the 
Moon  and  return  him  safely  to  Earth,  all  within  a  decade  (1961).  Just  three  months  shy  of  this 
deadline,  Apollo  1 4  landed  on  the  Moon,  with  Alan  Shepard  (the  first  American  in  space)  and  Edgar 
Mitchell.  Among  their  tasks  were  to  complete  two  extravehicular  activities  (EVA),  or  spacewalks, 
exploring  the  lunar  area  called  Fra  Mauro.  In  their  second  EVA,  their  destination  was  the  edge  of 
Cone  Crater,  an  impact  crater  in  the  vast  monochromatic  lunar  terrain.  Along  this  traverse,  the 
astronauts  became  uncertain  of  their  location  and  began  experiencing  fatigue  as  they  unexpectedly 
climbed  steep  terrain.  These  astronauts  searched  for  Cone  Crater,  wearing  bulky  spacesuits, 
breathing  hard,  hundreds  of  thousands  of  kilometers  away  from  Earth,  and  all  they  had  to  guide 
them  was  a  paper  map  (Figure  1.1). 


Figure  1.1  Apollo  14  astronaut  with  lunar  map  (NASA  image:  AS14-64-9089) 
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The  feat  of  sending  and  returning  humans  to  and  from  the  Moon  is  extraordinary.  A  new  national 
vision  for  human  space  exploration  has  been  set  forth  as  of  January  2004,  which  includes  returning 
to  the  Moon  and  going  beyond  to  other  planetary  surfaces  like  Mars.  Human  presence  on  planetary 
surfaces  will  involve  extra-vehicular  activities  (EVAs),  either  as  part  of  field  exploration, 
construction,  or  even  recreation.  In  these  futuristic  settings,  EVA  traversals,  be  it  on  foot  or  on 
rovers,  will  become  a  routine,  daily  task.  However,  in  light  of  the  Apollo  experience,  there  is  still 
research  and  development  that  can  be  done  to  improve  mission  planning,  productivity,  and  safety  of 
EVAs,  in  particular,  with  the  use  of  real-time  decision  support  systems.  Additionally,  we  can  take 
advantage  of  thirty-five  years  of  technological  advancements.  This  thesis  focuses  on  improving 
planetary  EVAs  through  human  and  automation  collaboration  by  investigating  the  real-time  task  of 
traversals  planning  and  re-planning. 

A  successful,  productive  planetary  exploration  program  must  take  advantage  of  the  strengths  of 
both  humans  and  automation,  be  it  computer  aids  or  robotics.  It  is  naive  to  consider  “robotic”  and 
“human”  missions  as  exclusively  one  or  the  other,  when,  in  fact,  all  missions  cannot  be 
accomplished  without  the  support  of  the  other: 

“. . .  in  the  space  program  it  has  been  common  to  consider  that  a  task  must  be  done  by  either 
an  astronaut  or  a  “robot”,  that  if  a  spacecraft  is  manned  then  astronauts  must  do  almost 
everything,  and  that  if  a  spacecraft  is  unmanned  every  task  must  be  automated.  In  fact,  on 
manned  spacecraft  many  functions  are  automatic,  and  on  unmanned  spacecraft  many 
functions  are  performed  by  human  remote  control  from  the  ground.”  (pg.  205,  Sheridan, 
2000) 

For  instance,  the  Mars  Exploration  Rovers  (MER)  are  two  robotic  agents  that  have  successfully 
explored  the  surface  of  Mars  for  almost  three  years  because  of  the  enormous  team  of  scientists  and 
engineers  operating  them  from  the  Earth.  It  is  this  assumption,  that  human-computer  collaboration 
will  be  integral  in  future  space  exploration  missions,  which  frames  this  thesis. 

Planetary  EVAs  are  complex,  hazardous  and  safety-critical.  EVA  tasks  currently  conducted  in 
microgravity  already  require  automation,  for  example,  for  life  support  system  control  and  remote 
manipulation  of  the  robotic  arm  for  positioning.  From  the  technological  perspective,  planetary 
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EVAs  require  and  depend  on  intricate  life  support  systems  for  operations  in  extreme  environments. 
Current  pressurized  space  suits  protect  astronauts  from  the  environment,  for  example,  providing 
thermal  regulation,  but  they  limit  mobility  and  senses,  e.g.,  sound,  vision,  smell  and  touch.  In 
addition,  there  is  a  limited  supply  of  consumables,  e.g.,  oxygen  and  battery  power.  Astronauts  must 
also  conduct  their  EVAs  within  prescribed  safety  constraints  and  operational  requirements. 

From  a  cognitive  decision-support  perspective,  a  traversing  astronaut  needs  to  manage  navigation, 
physiologic-  and  mission-specific  information,  all  in  time-finite  situations.  Planning  traversals  on 
other  planetary  surfaces  will  be  a  time  consuming  task  undertaken  weeks  and  months  before  the 
mission,  involving  many  scientists  and  engineers  who  recommend  a  feasible  path  while  keeping 
within  constraints  and  achieving  scientific  or  mission  goals.  Managing  all  this  information  becomes 
increasingly  difficult  when  the  task  changes  to  real-time  re-planning  of  a  traversal,  i.e.,  the  astronaut 
must  change  and  select  a  new  path  in  real-time  within  a  finite  amount  of  time  while  not  violating 
constraints.  Astronauts  will  have  many,  and  often,  competing  goals  (e.g.,  a  science  objective  versus  a 
safety  constraint)  for  which  they  will  have  to  find  an  acceptable  compromised  solution.  Re-planning 
of  an  EVA  is  inevitable  since  “the  unexpected”  is  inherent  to  exploration;  maximizing  mission 
success,  productivity,  and  safety  is  linked  to  the  astronauts’  and  mission  controllers’  ability  to  re¬ 
plan.  Hence,  a  cognitive  decision  support  system  is  would  be  essential  in  order  to  re-plan  an  EVA 
traversal  in-situ. 

Path  planning  and  re-planning  is  not  a  task  that  is  exclusive  to  EVA  traversals;  in  fact,  automated 
planners  are  becoming  ubiquitous,  and  are  integrated  into  daily  activities  such  as  driving.  Geospatial 
problem  solving  also  pertains  to  other  “moving”  objects,  be  it  a  soldier,  an  unmanned  vehicle  (air, 
ground,  and  underwater),  a  search  and  rescue  robot,  and  manned  aircraft.  Any  decision  support 
system  that  involves  these  objects  requires  the  human  operator  to  conduct  path  planning.  Thus, 
while  the  focus  of  this  thesis  is  on  human  space  exploration,  the  principles  examined  in  this  thesis, 
the  human  optimization  process  in  a  geospatial  task,  cut  across  all  these  mentioned  domains. 

1.2  Problem  Statement 

Traversal  re-planning  is  a  task  that  requires  knowledge-based  reasoning,  where  “faced  with  an 
environment  for  which  no  know-how  or  rules  for  control  are  available  from  previous  encounters  . . . 
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performance  is  goal-controlled  and  knowledge-based.”  (pg.  259,  Rasmussen,  1983).  Knowledge- 
based  reasoning  is  necessary  to  support  astronauts  confronted  with  uncertain,  novel  situations.  On 
the  other  hand,  mled-based  reasoning  is  emphasized  in  the  traditional  methods  of  astronaut  training, 
focusing  on  repeatedly  practicing  how  to  precisely  accomplish  a  specific  task  in  microgravity. 
Inevitably,  there  will  be  a  paradigm  shift  in  training  -  rule-based  training  will  not  be  sufficient  for 
future  space  exploration  missions.  Hence,  any  computer  decision  aids  must  support  knowledge- 
based  reasoning. 

The  central  research  question  of  this  thesis  addresses  what  is  the  appropriate  way  for  decisions 
support  systems  to  enable  knowledge-based  reasoning  in  geo-spatial  path  planning  problems.  In 
order  to  explore  this  problem,  different  human-automation  role  allocations  are  examined  and  tested 
in  order  to  understand  how  humans  conduct  complex  optimizations  under  different  automated 
assistance.  As  a  result,  the  aspects  of  human-computer  collaboration  that  promote  or  detract  from 
path  planning  performance  are  identified.  In  turn,  this  will  help  decision  support  system  designers 
leverage  human-automation  collaboration  to  enable  knowledge-based  reasoning. 

1.3  Thesis  Outline 

The  approach  to  exploring  human-computer  collaboration  in  a  path  planning  task  has  followed 
general  human-systems  integration  principles  for  the  design  and  testing  of  decision  support  systems 
for  planetary  exploration.  In  order  to  develop  a  path  planner,  the  operational  environment  needs  to 
be  first  defined.  This  was  accomplished  through  a  work  domain  analysis  of  human  planetary  EVAs. 
Functional  allocation  between  users  and  automation  are  determined,  i.e.,  different  automated 
assistance  in  support  of  users  in  decision  processes.  Human-in-the-loop  testing  is  employed  to 
understand  the  effects  of  the  automated  assistance  on  path  planning  performance.  Hence,  through 
experimentation,  effective  optimization  strategies  can  be  resolved.  Finally,  the  requirements  for 
decision  support  systems  for  real-time  EVA  planning  and  re-planning  can  be  generated  based  on 
principled  and  comprehensive  analysis.  This  thesis  addresses  each  of  these  steps. 

In  Chapter  2,  the  benefits  and  drawbacks  of  automation  are  described  and  relevant  past  path 
planning  studies  are  reviewed.  This  chapter  identifies  where  this  research  fits  in  the  overall  field  of 
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cognitive  systems  engineering  and  research  gap  it  addresses.  Guiding  hypotheses,  which  focus  on  the 
benefits  of  data  integration,  visualization,  and  role  of  sensitivity  analysis,  are  also  defined. 

Chapter  3  provides  a  both  a  historical  and  pragmatic  take  on  a  work  domain  analysis  for  the 
prototype  path  planner.  As  the  automation  planner  is  intended  for  future  lunar  and  Mars  traversals, 
one  cannot  “go  to  the  field”  and  study  how  planetary  exploration  in  extreme  environments  is 
performed.  Instead,  a  review  of  Apollo  EVAs  and  the  circumstances  involved  in  planning  and, 
more  importantly,  re-planning  of  lunar  sorties  completed.  This  overview  is  complemented  with  an 
observational  study  of  excursions  in  a  Mars-analog  site  in  the  Canadian  Arctic.  This  chapter 
concludes  by  presenting  a  new  Planetary  EVA  Framework  that  organizes  all  the  parameters 
pertaining  to  planning  and  re-planning  these  traversals. 

In  Chapter  4,  selected  parameters  and  constraints  from  the  Planetary  EVA  Framework  were 
incorporated  into  a  prototype  path  planner,  aptly  named  PATH  (Planetary  Aid  for  Traversing 
Humans).  This  chapter  describes  in  detail  all  the  components  of  PATH,  such  as  lunar  digital 
elevation  maps  and  the  implemented  cost  functions.  Most  significantly,  this  chapter  explains  the 
importance  of  the  developed  collaborative  visualization  termed  levels  of  equal  cost  (LOEC). 

Chapters  5  and  6  summarize  the  two  cornerstone  experiments  that  were  designed  and  implemented 
to  understand  how  humans  conduct  complex  optimization  under  various  automation  assistance  and 
collaborative  elements.  Experimental  hypotheses  are  delineated  in  each  chapter,  as  well  as 
independent  and  dependent  variables.  Results  are  presented  and  discussed  alongside  observed 
cognitive  strategies;  individual  experimental  conclusions  are  grouped  by  chapter. 

The  final  Chapter  7  consists  of  a  discussion  of  the  conclusions  of  this  thesis,  a  sensitivity  analysis 
meta-analysis  that  aggregates  results  from  both  experiments,  and  design  recommendations  for  path 
planning  and  re-planning  decisions  support  aids,  which  includes  a  real-time  EVA  aids.  Thesis 
contributions  are  listed  and  includes  framework  for  human-robotic  planetary  EVA  planning,  path 
planning  prototype,  quantification  of  path  planning  performance  across  various  conditions, 
identification  of  cognitive  strategies,  and  design  recommendations  for  planetary  EVA  decision 
support  aids. 
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2  Background  and  Hypotheses: 

Balancing  Humans  and  Automation 
Within  Decision  Support  Systems 


Balancing  task  allocation  between  humans  and  computers  is  crucial  to  the  development  of  effective 
decision  support  systems.  Within  types  of  decision  support  systems,  real-time  path  planning  and  re¬ 
planning  is  often  a  vital  task  which  involve  of  both  humans  and  computers.  There  are  many 
domains  that  exemplify  this,  such  as  military  and  commercial  pilots  planning  trajectories  during 
flight  or  ground-based  soldiers  constandy  revising  their  paths  over  the  ground  to  accommodate 
dynamic  changes  in  situations  and  resources.  Similarly,  unmanned  air  vehicle  (UAV)  operators  also 
re-plan  routes  in  response  to  emergent  threats  and  targets.  Robotic  applications  include  tele¬ 
operation  of  rovers  for  situations  like  search  and  rescue  missions  and  Mars  exploration  missions.  In 
these  cases,  human  geospatial  problem  solving  is  an  integral  part  of  the  operator’s  interaction  with 
the  automated  path  planners. 

Beyond  complex  decision  support  systems,  path  planning  has  become  part  of  everyday  life,  with  on¬ 
line  automated  planners  like  Mapquest1  and  Google  Maps2,  hand-held  GPS  (global  positioning 
systems),  or  in-car  navigation  systems.  The  alacrity  in  which  path  planning  algorithms  are  being 
integrated  into  technology  has  created  a  sudden  gap  in  our  understanding  of  how  people  interact 
with  automated  planners  and  how  different  aspects  of  automation  affect  path  planning  performance. 
While  methods  for  generating  paths,  or  trajectories,  for  autonomous  systems  have  been  extensively 


1  http:/ /www.mapquest.com 

2  http:/ /www.google. com/maps 
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studied  within  the  computer  science  field  in  order  to  develop  faster  and  more  robust  algorithms, 
little  attention  has  been  given  to  how  humans  optimize  trajectories  with  automated  assistance.  In 
terms  of  human-automation  interaction,  few  investigations  have  focused  on  how  people  optimize 
when  conducting  path  planning  in  high-risk,  time-critical  domains  that  include  a  certain  level  of 
uncertainty. 

The  domain  for  this  research  is  human  space  exploration,  which  embodies  the  definition  of  high- 
risk,  time-critical,  dynamic  and  uncertain  environmental  constraints.  It  is  under  these  circumstances 
that  astronauts  will  be  expected  to  path  plan  (and  re-plan)  in  order  to  optimize  their  extravehicular 
activities  (EVA)  on  other  planetary  surfaces.  With  such  a  large  problem  space  and  resultant 
complex  optimizations,  humans  will  have  to  leverage  automated  path  planners  automation. 

2.1  Benefits  and  Drawbacks  of  Automation 

Automation  is  generally  introduced  as  a  way  to  reduce  human  workload,  for  example,  by  automating 
manufacturing  processes.  As  computation  ability  has  increased,  automation  has  been  used  to  help 
humans  in  tasks  that  were  too  cognitively  intense  (such  as  too  many  variables  to  control  and/or 
excessive  computations).  For  example,  automation  is  essential  in  controlling  the  space  shuttle.  It  is 
also  used  to  help  track  and  manage  air  traffic.  However,  the  infusion  of  automation  has  changed 
people’s  responsibilities  and  work  from  active  participation  to  monitoring  and/or  supervising. 
Thus,  allocating  the  appropriate  role  to  humans  and  automation,  where  the  human  is  still  in  (or  on) 
the  loop  of  decision-making  has  been  the  focus  of  much  research  over  the  last  fifty  years. 

The  benefits  of  automation  were  succinctly  summarized  by  Fitts  in  the  1950s  (see  Table  2.1).  A 
computer’s  ability  to  be  repetitive,  fast,  and  precise  is  the  reason  why  automation  has  become  an 
integral  part  of  all  large,  complex  systems.  These  characteristics  also  make  it  ideal  in  the  use  of 
computer-generated  path  optimization,  when  the  states  of  all  variables  are  known  and  pre¬ 
determined.  In  juxtaposition,  a  human  has  unique  abilities  that  even  the  most  advanced  automation 
systems  still,  more  than  fifty  years  later,  do  not  have:  improvisation,  flexibility  and,  most  importantly, 
inductive  reasoning.  This  permits  humans  to  adapt  to  unexpected  circumstances,  resulting  in 
knowledge-based  reasoning,  the  type  of  problem  solving  in  which  humans  can  make  decisions  under 
novel  and  uncertain  situations  (Rasmussen,  1983).  Knowledge-based  behavior  (KBB)  is  goal-driven, 
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and  it  assumes  that  the  human  has  a  mental  representation  and  understanding  of  how  variables  in 
the  problem  space  are  interrelated.  Based  on  this  mental  model,  the  person  can  execute  deliberate 
actions  to  accomplish  a  goal,  instead  of  simply  following  a  set  of  predetermined  rules  (mle-based 
behavior). 


Table  2.1  Fitts’  list  (1951)  for  human-computer  role  allocation 


Humans  are  better  at: 

Computer  are  better  at: 

Perceiving  patterns 

Responding  quickly  to  control  tasks 

Improvising  and  using  flexible  procedures 

Repetitive  and  routine  tasks 

Recalling  relevant  facts  at  the  appropriate  time 

Reasoning  deductively 

Reasoning  inductively 

Handling  many  complex  tasks 

Exercising  judgment 

Fast  and  accurate  computations 

Automation  is  governed  by  coded  rules,  resulting  in  an  inflexibility  to  adapt  under  uncertainty, 
otherwise  known  as  automation  “brittleness”  (Layton,  Smith,  &  McCoy,  1994).  The  automation’s 
“brittleness”  in  a  highly  uncertainty  environment  can  be  mitigated  with  human  interaction  (Figure 
2.1).  A  similar  argument  exists  in  the  system  design  world,  where  under  changing  or  unknown 
system  and  environmental  factors,  a  flexible  engineering  design  is  desirable  (Saleh,  Hastings,  & 
Newman,  2003).  Thus,  for  decision  support  systems,  design  flexibility  is  analogous  to  human 
interaction  (Figure  2.2).  This  is  important  because  the  research  domain  of  this  thesis  is  space 
exploration,  where  “the  unexpected”  is  an  inherent  quality  of  the  domain.  Any  automated  path 
planning  aid  must  be  flexible  enough  to  support  astronauts’  knowledge-based  reasoning  while  taking 
advantage  of  the  strength  of  automation  computation  to  produce  adequate  path  solutions  in  high- 
risk  and  time-critical  domains. 
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Figure  2.1  Human  interaction  with  automation  as  a  function  of  certainty  (Cummings,  2006) 
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Figure  2.2  System’s  flexibility  and  robustness  (Saleh  et  al.,  2003),  adapted  to  include  human 

interaction 


Automation  “brittleness”  can  be  caused  by  the  models  (e.g.,  of  the  environment  or  process)  or  the 
algorithm  itself,  as  these  could  be  either  out-dated,  incomplete  or  incorrect.  Erroneous  models  are 
almost  inevitable  in  domains  in  which  any  uncertainty  exists.  However,  issues  related  to  automation 
use  go  beyond  the  automation  itself.  For  instance,  humans  could  fail  to  understand  the  automation 
and  the  computer-generated  solutions,  resulting  in  users  acquiring  incorrect  mental  representations 
of  the  way  the  automation  functions.  This  may  be  associated  with  the  automated  system’s  lack  of 
transparency,  where  models  are  inaccessible.  Even  when  automation  works  appropriately, 
researchers  have  found  evidence  for  inappropriate  knowledge  acquisition  (Glover,  Prawitt,  & 
Spilker,  1997)  and  skill  degradation  (Adelman,  Cohen,  Bresnick,  Chinnis,  &  Laskey,  1993).  When 
automation  fails,  users’  tmst  in  the  computer  aid  decreases,  which  could  lead  to  automation  not 
being  employed  (de  Vries,  Midden,  &  Bouwhuis,  2003;  Lee  &  Moray,  1994). 

The  use  of  automation  may  result  in  inability  to  maintain  mode  awareness  (Sarter  &  Woods,  1994), 
which  is  similar  to  the  degradation  of  situation  awareness  (SA)  reported  by  others  (Endsley,  1996; 
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Endsley  &  Kaber,  1999;  Strauch,  1997).  SA,  as  defined  by  Endsley,  is  (a)  the  perception  of  elements 
in  the  current  environment,  (b)  the  integration  and  comprehension  of  these  elements,  and  (c)  the 
projection  of  future  status  based  on  comprehension  (Endsley,  1995).  A  loss  of  SA  may  be  attributed 
to  the  human  operators’  lack  of  understanding  of  how  and  why  a  solution  was  generated  by  the 
automation.  SA  is  crucial  to  the  human  ability  to  apply  knowledge-based  reasoning  such  that  they 
can  develop  workaround  strategies  when  automation  fails  or  is  unreliable. 

Contributing  to  the  human’s  lack  of  situation  awareness  may  be  another  well-known  problem, 
automation  over-reliance  (Layton  et  al.,  1994;  Parasuraman,  Molloy,  &  Singh,  1993;  Parasuraman  & 
Riley,  1997).  Over-reliance  can  be  considered  a  symptom  of  automation  bias  (Cummings,  2004; 
Mosier,  Skitka,  Heers,  &  Burdick,  1998;  Skitka,  Mosier,  &  Burdick,  1999),  which  is  the  tendency  by 
users  (or  decision  makers)  to  disregard  or  not  search  for  contradictory  information  about  a 
computer-generated  solution.  As  a  result  of  automation  bias,  errors  of  commission  and  omission 
increase  (Mosier  &  Skitka,  1996;  Skitka  et  al.,  1999).  In  errors  of  commission,  the  decision  maker 
continues  acting  as  the  automation  dictates  even  though  there  is  evidence  that  the  solution  is 
erroneous.  In  errors  of  omission,  the  decision  maker  fails  to  intervene  because  the  automation  has 
not  indicated  errors. 

While  automation  has  many  drawbacks,  the  answer  is  not  necessarily  to  reduce  the  amount  of 
automation.  For  example,  within  the  domain  of  fault  management,  when  the  dynamics  of  a  fault  are 
fast,  automation  needs  to  be  used  to  respond  accordingly  (Moray,  Inagaki,  &  Itoh,  2000). 
Automation  is  indispensable  for  solving  these  time-critical  faults,  and  the  operator’s  role  is  still 
crucial.  Automation  is  necessary  within  all  decision  support  for  complex  systems,  particularly  when 
the  tasks  involve  a  large  problem  space  under  time  pressure.  Thus,  essential  to  developing 
automated  aids  for  planetary  exploration  is  the  balance  between  automation  assistance  and  human 
interaction,  while  mitigating  the  issues  previously  listed  (lack  of  automated  solution  transparency, 
loss  of  situation  awareness,  and  propensity  towards  automation  bias). 

2.2  Path  Planning  Task  and  Automation  Support 


With  respect  to  the  geospatial  problem  solving,  there  have  been  few  studies  examining  how 
cognitively  humans  optimize  paths  and  only  three  that  investigated  the  interaction  between  human 
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path  planning  performance  and  automated  assistance.  This  research  is  focused  on  the  planning  and 
optimization  of  paths,  i.e.,  before  execution  and  thus  actual  navigation  with  the  path  planner  is 
beyond  the  scope  of  this  thesis. 

When  no  automated  assistance  is  involved,  a  few  studies  have  investigated  path  planning 
optimization.  These  studies  focused  on  how  people  solve  a  traveling  salesperson  problem  (TSP), 
which  is  finding  the  shortest  path  given  a  certain  number  of  cities  (nodes)  with  the  constraint  of 
having  to  visit  each  city  once  (Graham,  Joshi,  &  Pizlo,  2000;  MacGregor,  Ormerod,  &  Chronicle, 
2000;  Pizlo  &  Li,  2003).  TSP  is  considered  to  be  a  non-deterministic  polynomial  (NP)  time 
complete  problem,  which  means  there  is  unlikely  an  algorithm  that  can  solve  the  problem  in  a 
polynomial  fashion1.  Graham  et  al.  (2000)  found  that  subjects  were  good  at  finding  close  to  optimal 
paths  for  when  the  distance  between  the  cities  were  equivalent  to  Euclidean  distance  (E-TSP).  Yet, 
when  the  distance  was  a  multiple  of  the  Euclidean  distance  (e.g.,  path  segments  were  weighted), 
subjects  did  poorly.  Their  results  with  respect  to  shortest  distance  paths  are  not  too  surprising  since 
the  problem  depended  on  the  human’s  intrinsic  ability  to  perceive  relative  size  (Gibson,  1979),  and 
hence,  determining  the  shortest  distance  between  points  is  a  relatively  easy  task  for  humans. 
Nonetheless,  this  research  confirms  that  when  the  path  planning  task  goes  beyond  finding  the 
shortest  distance,  it  becomes  a  more  difficult  problem  for  the  human. 

In  real  world  scenarios,  the  path  planning  task  typically  involves  the  integration  of  many  more 
variables  and  most  do  not  depend  on  the  paths’  visual  size  differentials.  The  cognitive  process  of 
path  planning  is  summarized  in  Figure  2.3.  First  the  human  must  acquire  external  information,  such 
as  environmental  factors,  variables  and  constraints,  as  each  possible  path  plan  depends  on  these 
numerous  elements.  This  information  then  needs  to  be  integrated  with  the  set  of  goals  that  a  path 
needs  to  meet,  e.g.,  reaching  a  destination  in  the  shortest  time.  The  relationships  between  variables 
are  defined  through  cost  models.  Cognitively,  these  models  may  be  heuristics  or,  in  the  case  of 
shortest  path,  they  are  perceptually  driven  (i.e.,  visual  size  differentials).  Information,  costs,  and 
goals  must  be  incorporated  in  order  to  create  a  path.  The  path  planning  process  may  be  iterative,  as 


1  Currently,  algorithms  solve  this  problem  in  exponential  times,  as  a  function  to  the  number  of 
nodes. 
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the  human  could  attempt  several  different  paths  that  might  satisfy  the  goals.  It  is  up  the  path 
planner  to  asses  each  solution  and  determines  if  it  is  the  best  plan  possible.  Depending  on  the  goals, 
a  path  plan  might  be  an  optimal  path  or  it  might  simply  meet  some  threshold  (path  cost  satisficing). 
A  path  plan  is  only  resolved  once  all  these  processes  occur. 


Figure  2.3  Cognitive  model  for  path  planning  task 

This  path  planning  cognitive  model  is  not  unlike  the  simplified  human  information  processing 
model  presented  in  Parasuraman,  Sheridan,  and  Wickens  (2000),  which  includes  sensory  processing, 
perception/working  memory,  decision  making  and  response  selection.  Sensory  processing  is  akin  to 
the  acquisition  of  information,  where  all  available  input  data  enters  for  processing.  The  human  path 
planner  then  must  use  his/her  working  memory  to  integrate  inputs  before  suggesting  a  path 
solution.  The  path  generation  process  is  thus  equivalent  to  the  decision  making  stage.  The  loop 
within  the  path  generation  stage  symbolizes  that  the  information  processing  is  not  a  strict  serial 
sequence  (also  acknowledged  by  Parasuraman  et  al.  2000),  but  rather  a  process  which  includes 
attempting  different  path  solutions  and  assessing  these  against  the  required  goals.  Finally,  the 
response  selection  stage  is  the  end  state  of  the  path  planning  cognitive  model,  where  a  path  solution 
is  determined. 

The  process  of  changing  (or  tweaking)  a  path  solution  (represented  by  the  loop  within  the  path 
generation  phase)  is  the  process  of  conducting  sensitivity  analysis  on  a  possible  path  solution.  For 
Saltelli  (2000),  sensitivity  analysis  “studies  the  relationship  between  information  flowing  in  and  out 
of  [a]  model.”  Essentially,  sensitivity  analysis  is  assessing  the  effect  of  changing  input  variables  of  a 
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model  or  solution.  Consequently,  during  path  planning,  it  is  the  process  of  modifying  a  path, 
evaluating  the  change  relative  to  the  goal,  and  iterating  until  a  satisfactory  solution  is  achieved. 
Pannell  (1997)  lists  decision  making  among  the  uses  of  sensitivity  analysis,  which  includes  testing  the 
robustness  of  an  optimal  solution,  investigating  sub-optimal  solutions,  and  identifying  sensitive  or 
important  variables.  Consequently,  when  the  human  path  planner  conducts  sensitivity  analysis, 
he/ she  is  learning  about  how  to  optimize  a  path  and  this,  in  turn,  may  lead  to  a  better  understanding 
of  how  a  solution  (in  this  case,  a  path  solution)  is  affected  by  changes  in  input  variables  (Saltelli  et 
al.,  2000).  Thus,  sensitivity  analysis  is  an  integral  part  of  the  path  planning  task. 

For  path  planning,  automation  can  be  used  in  different  parts  of  the  cognitive  process.  The 
automation  may  integrate  some  of  the  information,  like  variables,  environment  and  cost,  for  the 
human  so  that  he/ she  does  not  need  to  calculate  the  total  costs  of  the  path  solutions.  Another 
possible  example  is  that  automation  generates  path  solutions  and  the  human  selects  the  one  that 
meets  the  goals.  Wherever  automation  is  leveraged,  it  will  influence  human  path  planning 
performance.  A  few  studies  (described  subsequently)  have  investigated  various  methods  of 
leveraging  automation  in  the  path  planning  task  and  the  effects  on  human  planning  performance. 
These  have  focused  on  human  performance  within  the  domain  of  aviation  decision  making, 
specifically  investigating  path  planning  and  re-planning  of  flight  trajectories. 

Layton  et  al.  (1994)  developed  and  tested  a  prototype  en-route  flight  planner  which  required  pilot 
participants’  to  re-plan  an  aircraft  flight  trajectory  in  order  to  adapt  to  a  change  in  environmental 
conditions  (see  also  Smith,  McCoy,  &  Layton,  1997).  Three  versions  of  the  planner  were  explored, 
each  version  differing  in  the  amount  of  automation  assistance  provided.  The  lowest  automation  level 
possible  allowed  participants  to  “sketch”  flight  trajectories  on  the  computer  map,  i.e.,  the  human 
planner  sketched  paths  and  the  computer  filled  in  trajectory  details  such  as  arrival  times.  At  the 
highest  level,  the  computer  interface,  without  prompting  by  the  participant,  provided  a  solution  to 
the  flight  trajectory  conflict.  In  the  middle,  the  participant  could  ask  the  computer  to  provide  a 
flight  path  solution  based  on  selected  constraints.  The  main  result  was  that  participants  tended  to 
over-rely  on  the  computer-generated  solutions,  selecting  sub-optimal  paths.  A  possible  reason  was 
that  participants  did  not  explore  the  problem  space  as  much  when  presented  with  a  solution. 
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Chen  and  Pritchett  (2001)  evaluated  a  prototype  computer  aid  for  emergency  aircraft  trajectory 
generation  (for  divert  conditions),  simulating  both  plan  and  execution  of  new  trajectories  by  pilots. 
The  subjects’  task  was  to  create  a  new  trajectory,  under  an  emergency  scenario,  that  minimized  time 
to  land  and  did  not  violate  constraints,  i.e.,  weather,  airspace  regulations,  and  aircraft  limits.  The 
authors  tested  three  conditions:  re-planning  with  no  aid  (only  paper  charts),  with  an  aid,  and  an  aid 
with  pre-loaded  plans.  Performance  was  measured  by  time  to  land  and  workload.  The  worse 
performance  was  seen  in  the  condition  where  there  was  an  aid,  but  no  pre-loaded  plan.  The  best 
performance  was  achieved  when  pilots  adopted  the  pre-loaded  plans,  though  there  was  no 
significant  difference  with  the  test  condition  of  no  aid.  The  authors,  however,  favored  the  pre- 
loaded  aid  since  in  a  few  instances,  pilots  were  unable  to  create  and  evaluate  a  satisfactory  trajectory. 
This  study  also  presented  evidence  for  automation  bias,  as  some  pilots  did  not  improve  upon  the 
sub-optimal,  pre-loaded  plan,  as  well  as  over-reliance  that  occurred  when  pilots  accepted  erroneous 
information  presented  by  the  automated  aid. 

In  the  third  study,  Johnson  et  al.  (2002)  investigated  the  effects  of  time  pressure  and  automation  on 
a  route  re-planning  task.  Participants  were  asked  to  make  new  paths  that  would  first,  satisfy  mission 
constraints  (avoid  threat  areas  and  arrive  at  targets  within  an  acceptable  time-to-target  and  sufficient 
fuel),  and  second,  minimize  route  cost  (time  spent  in  hazard  zones  and  deviations  from  time-to- 
targets).  Six  time  pressures  were  imposed  on  the  scenarios,  ranging  from  20  to  125  seconds.  The 
automation  would  assist  the  participant  in  one  of  three  possible  ways:  1)  suggest  a  route  that  met 
mission  constraints,  2)  suggest  a  route  that  met  hazard  avoidance  rules,  and  3)  suggest  a  sub-optimal 
path  that  met  both  the  constraints  (time-to-target  and  fuel  restrictions)  and  avoided  hazard  zones.  A 
fourth  condition,  no  path  suggested,  was  also  tested  for  comparison.  Performance  was  measured  by 
route  cost  and  constraint  violation.  Based  on  their  experiment,  the  authors  concluded  that  full 
automation  was  most  beneficial  for  the  shortest  time  pressure  (less  than  30  seconds).  The  more  time 
participants  were  given,  performance  differences  decreased  between  automation  assistances.  The 
exception  to  this  trend  was  around  the  1  minute  mark,  when  participants’  performance  decreased 
with  more  automation  assistance.  There  was  also  evidence  for  complacency  and  automation  bias,  as 
noted  by  the  decreased  number  of  route  modifications  in  the  highest  level  of  automation. 
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While  humans  have  a  natural  capability  of  making  shortest  paths,  as  demonstrated  by  the  TSP- 
related  studies,  path  planning  in  real  world  scenarios  encompass  large  problem  spaces  that  may 
saturate  the  human’s  information  processing  capacity.  Automation  assistance  can  and  should  be 
leveraged  to  reduce  the  problem  space,  particularly  under  time  pressure.  The  studies  described 
above  chose  to  insert  automation  in  both  the  path  planning  processes  of  information  integration  and 
path  generation  (see  Figure  2.3).  Information  was  integrated  through  computer  aids  that  would 
calculate  path  costs  for  users,  such  as  arrival  times.  Path  plans  were  generated  for  the  humans,  be  it 
a  complete  or  a  partial  solution.  As  a  result,  the  human  had  to  assess  and  judge  if  the  path  solution 
met  the  goals  even  though  he/ she  had  not  partaken  in  generating  the  solution.  Thus,  these  studies 
indicate  that  complete  removal  of  human  operator  from  the  path  creation  process  leads  to 
complacency,  over-reliance,  and  automation  bias  issues. 

Even  though  sensitivity  analysis  is  an  integral  part  of  path  planning,  none  of  these  studies  focused 
on  how  the  automated  interface  may  have  affected  this  process.  In  fact,  how  humans  conduct 
sensitivity  analysis,  the  effect  on  human  performance,  and  how  best  to  support  this  process  within 
decision  support  systems  has  received  little  research  interest.  Most  of  the  literature  related  to 
sensitivity  analysis  centers  on  its  importance  for  model  development  and  validation  (e.g.,  Lu  & 
Mohanty,  2001;  McCarthy,  Burgman,  &  Ferson,  1995)  and  the  mathematical  methodology  of 
conducting  sensitivity  analysis  (e.g.,  Frey  &  Patil,  2002;  Saltelli  &  Bolado,  1998).  A  few  articles  have 
advocated  for  sensitivity  analysis  tools  for  multi-attribute  decision  making  aids  (e.g.,  Jimenez,  Rios- 
Insua,  &  Mateos,  2003;  Triantaphyllou  &  Sanchez,  1997).  However,  no  evaluation  of  the  specific 
tools  is  mentioned,  particularly  in  terms  of  human  performance. 

One  series  of  studies  has  investigated  interactive  optimization,  where  the  human  and  the  automation 
are  both  involved  in  an  optimization  task  (Anderson  et  al.,  2000;  Klau,  Lesh,  Marks,  Mitzenmacher, 
&  Schafer,  2002;  Scott,  Lesh,  &  Klau,  2002).  Their  description  of  interactive  optimization  is  similar 
to  sensitivity  analysis  because  the  researchers  have  tested  a  few  methods  that  allow  the  human 
operator  to  change  or  modify  solutions  (in  this  case,  path  solutions  for  vehicle  routing  with  time 
windows).  These  researchers  also  support  the  assertion  that  by  allowing  operators  to  guide  or  steer 
an  automatic  optimization  process,  operators  will  be  more  likely  to  understand  a  solution  they 
helped  create.  Even  though  the  experiments  include  less  than  a  handful  of  participants,  these 
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studies  have  shown  that  interactive  optimization  is  not  detrimental  in  the  optimization  process  and 
actually  surpasses  unguided  search  algorithms.  These  studies  also  point  to  different  strategies 
implemented  by  the  participants,  but  with  as  many  strategies  as  number  of  subjects,  trends  were  not 
assessed  (Scott  et  al.,  2002).  Clearly,  there  is  still  much  to  be  learned  about  how  best  to  support 
human  sensitivity  analysis  and  the  associated  possible  benefits. 

This  thesis  begins  to  address  the  role  sensitivity  analysis  has  on  human  path  planning  performance. 
Furthermore,  it  focuses  on  introducing  automation  in  the  path  planning  process  that  will  lead  to 
sensitivity  analysis,  and  hence  collaboration  between  the  human  and  the  decision  support  aid.  For 
instance,  in  the  previously  cited  studies,  either  a  path  was  automatically  generated  or  it  was  left  for 
the  human  operator  to  solve.  This  thesis  investigates  an  intermediary  state  in  the  path  generation 
stage  of  path  planning,  where  the  human  leverages  automation  to  create  a  path  solution,  allowing  for 
sensitivity  analysis  and  requiring  the  user  to  be  actively  involved  in  deciding  a  solution.  This  thesis 
also  examines  the  impact  of  aggregate  data  visualizations  for  multivariate  path  planning  problems. 

2.2.1  Levels  of  Automation 

The  previous  section  focused  on  allocating  automation  within  particular  stages  of  the  path  planning 
process.  More  generally,  functional  task  allocation  refers  to  the  amount  of  automation  used  in 
decision  support  systems  and  the  level  of  human  involvement.  For  example,  in  Chen  and  Pritchett’s 
computer  aid  (previously  discussed),  the  automation  was  allocated  the  task  of  generating  emergency 
aircraft  trajectories  while  the  pilot  either  accepted  or  modified  these  paths.  Several  frameworks  have 
been  proposed  to  describe  these  allocations,  typically  called  levels  of  automation  (LOA).  These 
frameworks  (subsequently  discussed)  help  in  the  design  of  a  path  planning  aid. 

The  most  commonly  cited  levels  of  automation  are  Sheridan  and  Verplank’s  (1978)  list  which  have 
evolved  (Parasuraman  et  al.,  2000)  to  a  list  of  ten  (Table  2.2).  Sheridan’s  LOA  range  from  one 
extreme,  where  the  human  makes  all  the  decisions  and  actions,  to  another,  where  only  the 
automation  decides  and  acts.  At  the  lower  levels,  the  human  takes  a  more  active  role  in  the  decision 
making  process,  from  finding  a  solution  (or  decision)  to  sorting  through  possible  alternatives 
suggested  by  the  computer.  At  level  5  (typically  called  management-by-exception),  the  automation 
begins  to  take  an  active  role  in  decisions,  and  subsequently,  the  human  is  only  required  to  accept  or 
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veto  solutions  presented  to  them.  At  the  higher  levels,  the  human  is  taken  out  of  the  decision 
making  loop  altogether. 


Table  2.2  Levels  of  automation  (from  Parasuraman  et  al.,  2000) 


1  The  computer  offers  no  assistance:  human  must  take  all  decision  and  actions. 

2  The  computer  offers  a  complete  set  of  decision/ action  alternatives,  or 

3  narrows  the  selection  down  to  a  few,  or 

4  suggests  one  alternative,  and 

5  executes  the  suggestion  if  the  human  approves,  or 

6  allows  the  human  a  restricted  time  to  veto  before  automatic  execution,  or 

7  executes  automatically,  then  necessarily  informs  humans,  and 

8  informs  the  human  only  if  asked,  or 

9  informs  the  human  only  if  it,  the  computer,  decides  to. 
^^^^^T^^^^^^^^^^The^om^uter^ecides^everythin^and^acts^autonomouslyA^ioringjheJiumarr 


The  previous  path  planning  studies  cited,  the  tested  range  of  LOA  tested  was  from  1  to  5.  For 
example,  Chen  and  Pritchett’s  (2001)  emergency  aid  could  be  described  as  having  LOA  4  since 
participants  requested  new  trajectories,  which  the  automation  supplied.  One  version  of  Layton’s  et 
al.  (1994)  aid  provided  a  trajectory  without  being  prompted  by  the  user,  akin  to  LOA  5.  It  is 
important  to  note  that  Sarter  and  Schroeder  (2001)  suggest  that  automated  decision  support  systems 
within  high-risk  domains  should  not  implement  LOA  5  as  users  may  become  vulnerable  to 
automation  biases. 

While  there  are  ten  levels,  this  set  of  LOA  fall  short  of  describing  many  systems.  If  the  lower  levels 
(2  through  4)  are  applied  to  the  topic  of  this  research,  human-computer  collaboration  within  path 
planning,  the  role  of  the  automation  is  essentially  restricted  to  one  or  many  paths  generated.  The 
biggest  limitation  of  Sheridan’s  LOA  is  that  there  is  no  mention  of  how  automation  can  assist  in  the 
analysis  phase  of  the  decision  making  process1. 


1  In  Parasuraman,  Sheridan  and  Wickens  (2000),  the  authors  do  acknowledge  that  a  similar  set  of 
LOAs  may  be  applied  to  information  acquisition,  information  analysis,  decision  selection,  and  action 
implementation.  However,  no  proposed  list  adapted  to  each  decision  process  step  yet  exists. 
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Two  other  levels  of  automation  frameworks  are  worth  mentioning:  Riley’s  and  Endsley  &  Kaber’s 
LOA.  Riley  (1989)  describes  a  taxonomy  of  automation  states  (see  Figure  2.4)  where  each  state  is  a 
combination  of  the  automation’s  role  and  level  of  information  it  can  process  (denoted  as 
intelligence).  Unfortunately,  while  proposing  a  greater  resolution  to  Sheridan’s  LOA,  it  remains 
vague  as  to  how  automation  is  to  fulfill  these  states  (e.g.,  what  must  the  automation  do  in  order  to 
be  an  assistant  level  of  autonomy  with  personalized  intelligence?).  The  general  trend  however  is 
highlighted  in  Figure  2.4,  where  higher  levels  of  intelligence  and  autonomy  are  desirable  with 
increasing  problem  space  complexity.  Endsley  and  Kaber  (1999)  propose  a  tendevel  taxonomy  of 
LOA  which  depends  on  four  generic  functions  (monitoring  system  status,  generating  options, 
selecting  option,  and  implementing  option)  that  could  be  allocated  to  either  human,  computer  or 
shared.  Each  of  the  ten  levels  is  some  combination  of  function  and  role  allocation  (Figure  2.5). 
This  taxonomy  may  be  adequate  to  describe  the  type  of  decision  support  system  implemented  as  a 
whole  (as  opposed  to  the  level  of  automation),  however,  it  is  still  vague  as  to  what  exactly  is  implied 
by  shared  (human/ computer)  allocation. 
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Figure  2.4  Riley’s  (1989)  automation  taxonomy,  adapted  to  include  trend  with  complexity 
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Figure  2.5  Levels  of  automation  by  Endsley  and  Kaber  (1999) 


These  LOA  frameworks  help  in  the  design  of  an  automated  path  planning  tool.  With  increasingly 
complex  problem  space,  more  intelligence  and  autonomy  is  necessary,  as  implied  by  Riley’s  LOA. 
Riley’s  framework  introduces  the  idea  that  a  higher  level  of  intelligence  is  more  than  presenting  raw 
data.  Furthermore,  there  is  a  specific  level  of  autonomy  for  “information  fuser”.  Thus,  a  path 
planning  aid  should  leverage  automation  to  integrate  raw  data  for  the  human  operator,  particularly  if 
the  task  is  complex.  The  simplest  method  of  integrating  information  for  path  planning  is  to 
calculate  the  paths’  cost.  Beyond  that,  visualizations  may  be  used  present  integrated  data.  In  terms 
of  path  generation,  based  on  Sheridan’s  LOA,  either  the  human  operator  or  the  automation  may 
generate  one  or  multiple  possible  path  solutions.  Present  among  the  levels  of  automation  by 
Endsley  and  Kaber  is  the  option  of  a  sharing  the  generation  of  solution  functionality,  i.e.,  both 
human  and  computer  participate  in  creation  of  path.  Thus,  a  path  planning  tool  could  be  designed 
to  accommodate  path  generation  by  the  human  operator,  by  the  automation,  or  through  a  shared 
method. 


2.3  Hypotheses 

In  general,  previous  studies  suggest  that  for  the  task  of  path  planning  and  re-planning,  higher  levels 
of  automated  assistance  was  beneficial  for  time  pressured  scenarios.  Yet  automation  bias  and  over¬ 
reliance  occurred  in  every  instance,  particularly  when  automated  solutions  were  sub-optimal.  While 
these  studies  focused  on  user  performance  with  different  LOAs,  they  did  not  attempt  to  quantify 
how  well  humans  optimized  under  different  task  complexities  or  conducted  sensitivity  analyses. 
Moreover,  there  was  no  discussion  of  the  impact  of  the  visualizations  used,  which  could  significantly 
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alter  the  results.  These  visualizations  integrate  information  the  human  operators  use  to  generate 
path  solutions.  A  primary  goal  of  this  research  is  to  examine  not  only  the  impact  of  increasing  the 
amount  of  automation  used  in  the  path  planning  task  on  human  performance,  but  also  investigate 
the  effects  of  visualizations  in  order  to  find  an  effective  method  of  leveraging  human-computer 
collaboration  with  respect  to  path  planning  optimization. 

This  thesis  addresses  two  main  guiding  hypotheses,  which  result  in  analysis  and  investigation  of 
different  aspects  of  human-computer  collaborative  path  planning  under  time-pressure,  their  impact 
on  performance  and  optimization  strategies,  across  increasingly  complex  problems.  The  more 
complex  tasks  include  degraded  automation  conditions  and  cost  functions  that  integrate  numerous 
variables. 

The  first  research  hypothesis  addressed  is: 

Hypothesis  /:  As  the  path  planning  becomes  more  complex,  e.g.,  an  increasing  number 
of  manipulated  variables  for  the  path  planning  task  and  degraded  automation 
conditions,  higher  amounts  of  data  integration  through  both  computation  and 
visualization  will  be  beneficial  for  the  human  operator.  These  benefits  include 
quickly  creating  near-optimal  paths,  maintaining  awareness  of  important  variables, 
and  compensating  for  imperfect  automation. 

Data  integration  through  computation  includes  leveraging  automation  to  generate  path  segments 
and  to  calculate  total  costs.  As  other  past  path  planning  studies  have  already  demonstrated 
problematic  issues  with  using  automation  solely  to  generate  path  solutions  (Chen  &  Pritchett,  2001; 
Johnson  et  al.,  2002;  Layton  et  al.,  1994),  this  research  will  focus  on  investigating  human-computer 
(shared)  path  creation.  Furthermore,  in  this  research,  data  integration  can  also  be  accomplished 
through  a  visualization  that  aggregates  information  (both  variables  and  costs)  based  on  the  path 
planning  algorithm  used  to  search  for  least-costly  paths.  Thus,  this  visualization  may  simplify 
complex  path  planning  problems  for  the  human  operator. 
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The  second  research  hypothesis  investigated  is: 


Hypothesis  2\  When  a  computer  system  that  promotes  sensitivity  analysis  is  used  for 
the  task  of  planning  (and  hence,  re-planning),  human  path  planning  performance  will 
increase.  Improvements  include  providing  the  human  operator  the  ability  to 
understand  how  to  optimize  paths,  thus  achieving  near-optimal  solutions. 

No  research  to  date  has  explored  the  role  sensitivity  analysis  plays  in  the  task  of  path  planning  in 
terms  of  human  performance,  even  though  it  is  an  integral  part  of  the  human  optimization  process. 
Sensitivity  analysis  is  the  process  of  observing  how  much  a  solution  changes  as  input  variables  are 
modified.  Conducting  sensitivity  analysis  is  a  more  time-consuming  optimization  strategy,  however, 
computational  power  can  be  leveraged  to  provide  aids  (usually  called  “what-if”  tools)  for  the  human 
operator  that  will  assist  in  sensitivity  analysis.  This  can  be  accomplished  by  making  path 
modifications  and  corresponding  evaluations  easily  available.  If  the  human  operator  conducts 
sensitivity  analysis,  he/ she  will  begin  to  acquire  a  better  understanding  of  how  to  fundamentally 
optimize  paths,  which  is  particularly  beneficial  as  the  task  becomes  increasingly  complex.  As 
discussed  before,  task  complexity  may  be  a  due  to  number  of  variables  manipulated  during 
optimization  process  or  degraded  automation  conditions.  For  the  latter,  an  increase  in  human  path 
planning  performance  will  be  observed  if  the  human  operator  successfully  adapts  path  planning 
strategies  and  is  able  to  quickly  create  near-optimal  paths.  Finally,  as  the  role  of  sensitivity  analysis 
will  be  concurrently  investigated  with  different  functional  allocations  for  path  generation  and 
visualizations,  potential  cross-effects  can  be  identified. 

In  order  to  investigate  these  guiding  hypotheses,  an  automated  path  planner  was  developed  and 
modeled  as  a  planning  computer  aid  for  astronauts  conducting  extravehicular  activities  on  the 
surface  of  the  Moon.  The  prototype  was  built  based  on  a  work  domain  analysis  of  both  Apollo 
historical  evidence  and  observational  study  of  excursions  at  a  Mars-analog  site.  Two  separate 
experiments  were  designed  and  tested,  addressing  the  research  questions  presented  in  this  chapter. 
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3  Work  Domain  Analysis 


Characterizing  Surface  Operations 


3.1  Extravehicular  Activities  on  Planetary  Surfaces 

During  the  Apollo  program,  exploration  was  limited  to  a  maximum  3  day  stay,  with  two  astronauts 
in  bulky  spacesuits,  and  the  use  of  a  lunar  rover  in  later  missions.  Astronauts  explored  areas  that 
were  within  a  few  kilometers  of  the  lunar  module.  NASA  is  now  considering  lunar  missions  where 
up  to  four  astronauts  are  traversing  simultaneously  on  the  surface  of  the  Moon,  taking  advantage  of 
pressurized  or  un-pressurized  rovers  (NASA,  2005).  Future  stays  on  the  lunar  surface  will  range 
from  seven  days  to  180  days,  with  up  to  four  sorties  per  week;  Mars  missions  may  last  up  to  600 
days.  As  astronauts  stay  longer  on  other  planetary  surfaces,  astronauts  will  be  responsible  for 
planning  and  re-planning  of  their  own  extravehicular  activities  (EVAs). 

In  order  to  design  robust  decision  support  aids  for  mission  planning,  the  first  step  is  to  characterize 
surface  operations  of  planetary  exploration.  This  assessment  is  called  a  work  domain  analysis 
(WDA)  because  this  investigation  attempts  to  describe  the  structure  of  the  domain,  e.g.,  a  planetary 
EVA,  in  terms  of  not  just  the  tasks  that  are  expected  to  be  performed  but  also  the  constraints  and 
impact  of  the  environment.  The  goal  of  this  WDA  is  to  understand  and  identify  the  information 
requirement  for  a  path  planning  decision  support  system.  The  work  domain  analysis  presented  in 
this  chapter  does  not  adhere  to  the  rigid  WDA  definition  (i.e.,  an  abstraction  hierarchical 
decomposition)  presented  in  Rasmussen  (1985)  nor  is  it  part  of  a  formal  cognitive  work  analysis 
(Vicente,  1999).  The  approach  subscribed  to  is  more  general,  focusing  on  identifying  what 
information  is  required  for  planning  an  EVA  by  taking  advantage  of  multiple  methods,  including 
field  observations  and  review  of  documentation,  elements  that  are  more  characteristic  of  a  cognitive 
task  analysis  (Rasmussen,  Pejtersen,  &  Goodstein,  1994;  Schraagen,  Chipman,  &  Shalin,  2000). 
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This  work  domain  analysis  is  not  a  trivial  task  as  future  surface  operations  may  consist  of  advanced 
technologies  (e.g.,  robotic  assistants),  different  objectives  (e.g.,  acquiring  in-situ  materials),  and  a 
growing  body  of  knowledge  (e.g.,  new  areas  of  scientific  value).  There  are  many  likely  combinations 
that  depend  on  the  infrastructure  and  mission  goals.  Hence,  a  myriad  of  complex  extravehicular 
activities  (EVA)  architectures  are  possible.  Instead  of  assuming  one  architecture,  this  chapter 
introduces  a  framework  that  captures  the  key  parameters  and  constraints  that  define  a  single 
planetary  EVA.  Through  the  work  domain  analysis,  planetary  EVA  variables,  parameters  that 
determine  and  affect  path  planning  and  re-planning,  are  identified  and  subsequently  described. 
These  variables  are  then  categorized  and  organized  into  the  Planetary  EVA  Framework.  Thus,  the 
framework  is  a  summary  of  the  work  domain  analysis,  which  was  conducted  by  reviewing  Apollo 
EVAs,  performing  an  observational  study  of  excursions  in  a  Mars-analog  site,  and  interviewing 
subject-matter  experts,  such  as  planetary  scientists  and  aerospace  engineers. 

3.2  Past  EVAs:  Apollo  Program 

3.2.1  Apollo  Extravehicular  Activities 

Every  Apollo  mission  that  landed  on  the  Moon  included  at  least  one  EVA.  During  the  Apollo  1 1 
and  12  missions,  the  EVAs  focused  more  on  engineering  testing,  while  the  EVAs  performed  during 
Apollo  14  -  17  missions  were  geared  more  towards  exploration  that  addressed  scientific  hypotheses. 
Extensive  preparation  was  undertaken  in  order  to  maximize  scientific  return,  involving  engineers, 
scientists,  astronauts,  and  mission  planners.  Routes  and  estimated  travel  times  were  established 
using  low-resolution  photographic  images  and  crude  topographic  maps.  The  team  of  scientists  and 
mission  planners  also  allocated  finite  times  to  the  scientific  tasks  (Muehlberger,  1981).  Each  site  and 
task  was  prioritized  based  on  its  relative  importance  to  the  overall  scientific  mission  goal.  Many 
activities  or  sites  were  dropped  due  to  the  lack  of  time,  which  is  the  underlying  recurring  theme 
among  the  lunar  EVAs. 

Some  estimates  were  poor  for  the  planned  EVA  tasks  and  thus,  required  additional  time,  affecting 
the  rest  of  the  schedule.  During  the  first  mission  that  astronauts  walked  on  the  lunar  surface 
(Apollo  11),  EVA  preparation  times  were  optimistic.  Donning  the  spacesuit  took  longer  than 
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expected,  delaying  the  start  of  the  first  EVA  on  the  Moon  (NASA,  1969).  Delayed  start  times 
reduced  the  length  of  later  EVAs,  as  was  the  case  in  Apollo  15  and  16.  Since  these  missions  landed 
on  the  Moon  later  than  expected,  sleep  periods  were  rearranged  so  that  astronauts  would  be  rested 
when  on  the  lunar  surface  (Jones,  1995),  reducing  the  overall  surface  “work”  time.  Occasionally, 
deploying  equipment,  like  the  Lunar  Rover  Vehicle  (LRV),  was  time-consuming,  or  worse,  the 
equipment  failed  or  worked  inappropriately.  For  example,  drilling  on  the  Moon  was  difficult  and 
took  longer  than  expected  for  astronauts  on  Apollo  15  and  17.  Any  additional  time  beyond  what 
was  budgeted  to  accomplish  tasks  collectively  resulted  in  changing  the  EVA. 

The  longest  “walking”  EVA1  occurred  during  Apollo  14,  where  astronauts  had  to  reach  Cone 
Crater,  a  destination  they  never  quite  reached  (Johnston  &  Hull,  1975;  NASA,  1971;  Waligora  & 
Horrigan,  1975).  With  only  a  paper  map  in  hand,  encumbered  by  a  bulky,  life  support  system, 
astronauts  had  to  traverse  an  unfamiliar  environment,  unaware  of  their  deviations  from  the  planned 
path.  Astronauts  had  poor  situation  awareness  of  their  location  that  resulted  from  inadequate 
surface  contrast,  lighting  conditions,  and  the  monochromatic  terrain  (Figure  3.1).  In  addition, 
astronauts  traversed  steep  terrain  that  resulted  in  high  metabolic  rates  and  increased  heart  rates, 
requiring  extra  rest  stops.  Having  fallen  behind  on  their  schedule  and  unable  to  accurately 
determine  the  edge  of  Cone  Crater,  their  true  destination  site  was  abandoned  and  the  astronauts  had 
to  settle  for  another  site. 


1  The  total  average  distance  traversed  was  1.5  km. 
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Figure  3.1  Edge  of  Cone  Crater,  Apollo  14  (NASA  image,  AS-14-64-9103) 

The  later  Apollo  missions  included  a  rover  (Lunar  Roving  Vehicle,  LRV),  which  allowed  astronauts 
to  traverse  longer  distances,  but  additional  concerns  arose  (Figure  3.2).  Apollo  16  astronauts 
commented  on  driving  slower  because  low  sun  angles  made  it  difficult  to  estimate  sizes  (e.g.,  crater 
sizes)  and  distances  (Jones,  1995).  The  LRV  was  yet  another  piece  of  equipment  that  experienced 
technical  difficulties,  such  as  a  “down”  navigation  system  (Apollo  16)  and  a  lack  of  power  steering 
(Apollo  15). 


Figure  3.2  Lunar  Roving  Vehicle  (LRV,  NASA  image,  AS-17-147-22526) 
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The  introduction  of  the  rover  to  the  EVAs  imposed  a  strict  safety  rule:  the  “walk-back” 
requirement.  If  the  LRV  became  inoperable,  the  contingency  plan  was  for  astronauts  to  walk  back 
to  the  lunar  module  (LM).  However,  astronauts  could  potentially  run  out  of  oxygen  before  reaching 
the  LM  if  it  was  too  far.  Therefore,  the  distance  between  the  rover  and  the  LM  was  limited  by  the 
astronaut’s  ability  to  walk  back  to  the  LM.  Due  to  the  “walk-back”  requirement,  the  farthest  sites 
had  the  most  time  pressure.  This  was  the  case  during  Apollo  17,  when  astronauts  found  “orange 
soil”,  which  led  to  collecting  an  unplanned  core  sample  (Jones,  1995).  The  crew  had  to  quickly 
assess  if  they  could  accomplish  the  additional  task  given  the  time  limitations.  They  chose  to  take  the 
core  sample,  however,  they  used  up  all  their  time  for  that  site  and  almost  violated  the  “walk-back” 
restriction. 

Finally,  additional  environmental  constraints  affected  general  performance,  and  thus,  efficiency. 
During  Apollo,  the  two  significant  environmental  factors  were  lighting  conditions  (in  conjunction 
with  the  lack  of  atmosphere)  and  dust.  For  example,  Apollo  12  astronauts  mentioned  lighting 
conditions  that  made  it  difficult  to  properly  assess  rock  samples  (NASA,  1970).  The  pervasive  lunar 
dust  affected  all  equipment,  resulting  in  deteriorated  performance.  Space  suits  were  very  dusty  after 
only  three  uses,  in  most  cases.  Time  spent  cleaning  equipment  during  an  EVA  added  another, 
though  small,  delay. 

3.2.2  Implications  of  Apollo  Experience 

The  successful  completion  of  each  planetary  EVA  during  Apollo  is  a  great  feat.  However, 
regardless  of  the  many  hours  spent  meticulously  planning  and  practicing  each  sortie,  every  single 
EVA  conducted  in  Apollo  had  to  do  some  sort  of  re-planning  in  order  to  adjust  to  the  unexpected. 
Over  the  course  of  one  sortie,  even  small  problems  accumulate  over  time,  requiring  astronauts  and 
mission  controllers  to  adapt  to  the  new  time  constraints.  There  are  many  incidents  that  resulted  in 
time  delays,  the  most  prominent  being  underestimations  of  time  required  to  complete  tasks. 

Table  3.1  summarizes  the  key  variables,  parameters  that  determine  and  affect  planetary  EVAs  path 
planning  and  re-planning,  identified  through  the  review  of  Apollo  excursions.  These  variables  make 
up  the  specifics  of  the  Planetary  EVA  Framework  (discussed  in  section  3.5).  Most  importantly,  the 
Apollo  experience  reveals  that  there  is  a  complex  relationship  between  these  variables.  For  instance. 


45 


the  interaction  between  spacesuit,  the  terrain,  and  mobility  was  poorly  understood,  which 
consequently  resulted  in  a  very  conservative  “walk-back”  requirement.  There  are  still  many 
unresolved  issues  left  from  the  Apollo  era,  such  as  navigation  and  perception  problems  due  to  the 
Moon’s  environment.  A  better  understanding  of  the  relationship  between  variables  will  improve 
upon  future  safety  and  science-return  of  Lunar  exploration. 

Table  3.1  Planetary  EVA  variables  based  on  Apollo  EVAs 


Variable  Category 

Variable  Specifics 

Astro-agent 

Astronaut 

Goals 

Sites  (number  and  locations),  prioritization  of  sites  and  tasks,  tasks  at 
sites,  costs  associated  with  task  (including  time,  energy,  learning  curve), 
additional  unexpected  tasks 

Terrain 

Characteristics  (craters,  large  rocks,  high  slopes)  related  to  obstacles 

Exploration  costs 

Distance,  time  traversing,  metabolic  (or  energy)  expenditures,  oxygen 
(and  consumables)  used,  favorable  sun  positions 

Transportation 

Walking,  driving;  preparation  time  (spacesuit  and  rover);  interaction 
between  transportation  and  terrain  &  environment 

Environment 

Sun  glare,  illumination  as  affecting  perception  and  navigation;  dust 

Infrastructure 

Lunar  lander,  access  to  lunar  rover 

Consumables 

Life  support  consumables  specific  to  astro-agent  (e.g.,  oxygen,  COa 
scrub,  water,  battery  supply) 

Operation  rules 

Safety  margins  (e.g.,  minimum  remaining  reserve  oxygen),  walk-back 
requirement,  operationally  required  rest  stops,  maximum  metabolic  rate 
&  heart  rate 

EVA  components 

Path  (sites  visited,  path  segments),  schedule  (time  at  site  and  traversing 
to  waypoints),  energy  consumed  along  path  segments 

3.3  Present  EVAs:  MER  and  Terrestrial  Analogs 

3.3.1  Planetary  Exploration  by  Mars  Exploration  Rovers  (MER) 

Mars  exploration  is  currently  conducted  on  a  daily  basis  by  the  two  operational  NASA  Mars 
Exploration  Rovers  (MER).  With  respect  to  planning,  a  group  of  scientists  and  engineers  meet 
every  Martian  day  (sol)  to  decide  on  the  sequence  of  commands  for  each  rover  (NASA-JPL,  2005a, 
2005c).  These  generally  relate  to  where  to  go,  what  activities  to  conduct  (e.g.,  images  to  take),  and 
where  to  position  the  rovers’  arm  (Biesiadecki,  Leger,  &  Maimone,  2005).  Decisions  are  made  based 
on  the  previous  sol’s  information,  which  includes  not  only  imagery  but  also  engineering  information, 
such  as  the  actual  position  and  energy  consumed  by  the  rovers.  If  activities  did  not  materialize  as 
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expected,  engineers  have  to  assess  the  reasons  why  and  adapt  to  reflect  the  discrepancies.  Mission 
planners  typically  allot  one  day  (8-10  hours)  to  develop  their  plan. 

The  overall  goals  of  the  daily  traverses  are  scientifically  driven.  This  means  that  site  selection  and 
activities  are  identified  by  the  science  leaders.  Exploration  sites  are  determined  based  on  imagery 
and  spectroscopy  data,  and  rely  on  planetary  scientists’  judgment  for  subsequent  site  selection.  The 
sites  that  each  of  the  rovers  have  investigated  were  not  pre-determined  on  Earth,  except  the  landing 
location.  All  other  sites  were  selected  in-situ,  and  were  resolved  based  on  previous  findings,  as  is  the 
nature  of  exploration  (Zuber,  2006). 

In  order  to  interact  with  the  rovers,  scientists  and  engineers  (on  the  ground)  use  the  Scientific 
Activity  Planner  (SAP)  (Norris,  Powell,  Fox,  Rabe,  &  Shu,  2005;  Norris,  Powell,  Vona,  Backes,  & 
Wick,  2005).  SAP  is  the  primary  science  operations  tool  that  provides  planning  capabilities  for  the 
MER.  Scientists  and  engineers  have  access  to  imagery  that  includes  panoramas,  camera  views,  and 
hyperspectral  visualizations.  Processed  information  includes  “reachability”  maps  (relating  to  where 
the  instruments  can  reach),  slope,  and  solar  energy  maps  (Leger,  Deen,  &  Bonitz,  2005).  After 
locations  of  interest  are  marked,  a  simulation  predicts  rover  details  like  the  power  consumed,  the 
data  volume,  time  required,  and  final  position  (Norris,  Powell,  Fox  et  al.,  2005). 

Key  to  the  rovers’  ability  to  conduct  exploration  is  their  navigation  ability.  The  MERs’  tele¬ 
operators  define  waypoints  and  end  goal  states.  The  rover  then  can  assess  and  determine  the  actual 
path,  avoiding  obstacles.  However,  defining  an  100-meters  drive  sequence  may  take  an  operator 
between  2  to  4  hours  (Biesiadecki  et  al.,  2005).  This  is  due  to  the  manual  terrain  analysis  that  is 
incorporated  in  the  process.  There  are  many  terrain  properties  that  are  important:  slope,  elevation, 
rock  size  &  position,  terrain  quality  (e.g.,  sand,  firmness),  and  homogeneity  of  terrain.  Many  of  these 
influence  how  the  rovers  traverse  including  speed,  slippage,  and  power  required.  If  the  rover 
encounters  a  navigation  problem,  the  operator  intervenes.  For  example,  one  of  the  rovers 
inadvertently  buried  itself  into  a  sand  dune  (Biesiadecki  et  al.,  2005;  NASA-JPL,  2005b).  The  rover 
followed  its  driving  commands  until  it  realized  that  it  was  no  longer  advancing.  At  this  point,  the 
rover  went  into  a  safe  mode  (“alive”  though  waiting  for  new  instructions),  alerting  the  engineers  of 
the  problem.  After  weeks  of  terrestrial  testing  drive  configurations  and  strategies,  engineers  were 
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able  to  command  the  rover  out  to  safety.  Such  an  incident  highlights  the  importance  of  operators, 
particularly  in  dealing  with  terrain  uncertainty. 


Assessment  of  on-going  MER  planetary  EVAs  identified  additional  variables,  beyond  the  Apollo 
experience,  that  need  to  be  taken  into  account  by  a  comprehensive  path  planner  (Table  3.2).  Future 
human  exploration  will  likely  be  similar  to  that  conducted  by  MER,  where  sites  of  interest  are  based 
on  recent  discoveries,  requiring  an  up-to-date  database  of  imagery,  spectral  data,  and  terrain 
properties.  Most  importantly,  the  experience  with  MER  highlights  the  value  of  operator  and  rover 
interaction.  In  this  case,  it  is  the  human  that  intervenes  when  the  robotic  agent  is  affected  by 
unexpected  events,  such  as  the  Martian  terrain  uncertainty.  In  turn,  future  robotics  agents  could 
also  prevent  humans  from  making  mistakes. 


Table  3.2  Planetary  EVA  variables  based  on  MER  exploration 


Variable  Category 

Variable  Specifics 

Astro-agent 

Robotic  agent  (tele-operated  rover);  capabilities  of  agent  (instrument- 
or  task-related) 

Goals 

Determined  by  on-going  exploration 

Terrain 

Rock  density,  ground  bearing  strength,  homogeneity,  slope,  elevation, 
spectral  data 

Transportation 

Minimum  ground  bearing  strength,  maximum  traversable  slope, 
maximum  speed  (function  of  terrain),  power  requirements,  admissible 
slippage 

Environment 

Wind;  seasonal  solar  supply 

Consumables 

Power  supply/ source,  life  cycles  of  robotic  agents’  components 

3.3.2  Planetary  Exploration  Studies  Using  Terrestrial  Analogs 

Two  studies,  Carr  (2001)  and  Clancey  (2001),  have  used  terrestrial  analogs  to  investigate  human 
planetary  exploration,  specifically  focusing  on  EVA.  While  Carr  emphasized  trades  between 
transportation  and  communication  ability  for  lunar  exploration,  Clancey  focused  on  information 
flow  between  explorers  when  planning. 

Carr  (2001)  examined  the  effects  of  distributed  architectures  for  Mars  surface  exploration.  There  are 
two  major  components  of  this  research  that  pertain  to  planetary  EVAs:  a  month-long  field 
geological  mapping  project  that  was  placed  into  the  context  of  planetary  surface  exploration,  and  an 
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in-depth  analysis  of  Apollo  EVA  traverse  to  Cone  Crater.  From  Inis  field  experience,  he  emphasized 
the  importance  of  the  providing  the  explorer  with  the  flexibility  to  alter  traversals,  particularly  as 
mission  goals  change  over  time.  He  argued  that  initially  human  missions  to  Mars  will  resemble 
Apollo,  with  pre-determined  sites  of  interest;  however,  with  longer  mission  durations,  new  sites  of 
interest  will  be  added.  This  strategy  shift  is  similar  to  the  on-going  MER  exploration.  Thus, 
traverses  later  in  the  mission  will  be  focused  on  solving  new  problems  (Carr,  2001).  In  the  Cone 
Crater  EVA  analysis,  Carr  identified  key  variables  for  planning  excursions:  visibility,  transportation 
costs  (including  metabolic  expenditures)  between  waypoints,  “accessibility  map”  or  “reachability 
map”,  and  communication  ability  with  ‘home’  mission  controllers  (Carr,  2001;  Carr,  Newman,  & 
Hodges,  2003). 

Clancey  (2001)  conducted  an  ethnographic  study  of  activities  within  the  Haughton-Mars  Project 
(HMP)  research  camp,  a  Mars-analog  site1.  His  main  objective  was  to  understand  how  astronauts 
may  live  and  work  on  Mars.  With  regards  to  excursion  planning,  he  described  it  as  a  group  activity, 
involving  assessment  of  multiple  representations,  such  as  satellite  and  aerial  imagery  of  the  area. 
The  scientists  felt  that  time,  date,  explorers,  location  of  exploration,  and  purpose  of  exploration 
were  the  key  descriptors  of  the  field  excursion.  Clancey  noted  differences  between  the  scientists  in 
the  way  they  chose  to  explore,  particularly  biologists  and  geologists.  Biologists  and  geologists  rarely 
traveled  together  as  they  had  few  sites  in  common,  and  biologists  visited  fewer  sites  than  geologists. 
Clancey  concludes  that  biologists’  search  is  “depth-first”  while  a  geologists’  search  is  “breadth-first”. 

Based  on  these  terrestrial  analog  studies,  additional  critical  planetary  EVA  variables  that  affect  path 
planning  are  summarized  in  Table  3.3.  Both  Carr  and  Clancey  emphasize  the  nature  of  human 
exploration,  where  current  information  leads  to  new  sites  of  interest  and  how  explorers  collaborate 
to  come  up  with  a  plan.  These  reports  overlap  with  the  already  presented  Apollo  and  MER  review, 
hence  only  few,  new  variables  are  listed.  Nonetheless,  these  studies  establish  a  method  for  using 
terrestrial  analogs  a  means  for  acquiring  operational  requirements  for  future  planetary  EVAs. 


1  Haughton-Mars  Project  site  is  discussed  more  in  a  subsequent  section. 
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Table  3.3  Planetary  EVA  variables  based  on  terrestrial  analog  studies 


Variable  Category 

Variable  Specifics 

Astro-agent 

Expertise  of  astronaut 

Goals 

Flexibility  to  add  new  sites 

Exploration  costs 

Visibility  (line  of  site),  reachability 

Infrastructure 

Communication  infrastructure  (e.g.,  beacons) 

3.3.3  Observational  Study:  Mars-Analog  Excursions 


In  support  of  this  thesis  and  to  further  investigate  future  planetary  EVAs,  an  observational  study  of 
excursions  was  conducted  at  the  Haughton-Mars  Project  (HMP),  a  terrestrial  Mars-analog  research 
site  set  in  an  uninhabited  island,  high  in  the  Canadian  Arctic.  HMP  is  sponsored  by  NASA  and  the 
Canadian  Space  Agency  (CSA)  and  managed  by  the  Mars  Institute.  They  host  an  international, 
interdisciplinary  group  of  scientists  and  engineers  who  return  yearly  within  the  operational  season,  a 
period  of  5  to  6  weeks. 

The  site  is  located  next  to  Haughton  Crater  (Figure  3.3),  a  38  million  year  old  impact  crater  on 
Devon  Island  (75°N,  90°  W).  The  area  is  of  interest  due  to  the  geological  similarities  to  the  Martian 
terrain.  Due  to  the  remoteness  of  this  Mars-analogue  terrain,  exploration-like  activities  can  be 
undertaken  by  geologists  and  other  scientists.  Thus,  excursions  conducted  at  this  site  resemble 
sorties  of  long-duration  planetary  space  missions.  Assessing  and  reviewing  HMP  excursions  can 
provide  insight  to  the  potential  problems  astronauts  may  face  in  future  planetary  EVAs. 


Figure  3.3  Haughton  Crater,  Devon  Island,  Canada.  Area  of  crater  excursion  highlighted. 
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During  the  2005  season1,  eight  excursions  (Table  3.4)  were  categorized  through  an  “EVA  Log”. 
Excursions  were  varied  as  researcher  at  the  camp  had  objectives  they  wanted  to  accomplish  within 
the  time  they  were  stationed  at  HMP.  For  example,  one  observed  traversal  was  conducted  by  two 
geologists  on  foot  to  a  near-by,  known  location  while  another  included  a  large  group  on  all-terrain 
vehicles  (ATVs)  to  an  unexplored  destination.  Some  traversals  observed  included  researchers 
wearing  a  mock  space  suit,  provided  by  Hamilton  Sundstrand.  Creating  an  “EVA  Log”  permitted  a 
systematic  assessment  of  each  excursion. 

Using  the  log,  each  excursion  was  documented  through  a  combination  of:  a)  observation  of  pre¬ 
excursion  planning  session  and  post-excursion  debriefings,  b)  pre-  and  post-excursion  interviews 
with  excursion  participants  in  addition  to  observations  taken  from  actual  excursions.  Multiple 
methods  were  implemented  in  order  to  best  capture  as  many  excursion  experiences  as  possible 
without  interfering  with  preparation  for  and  execution  of  the  excursions.  Aside  from  the  “EVA 
Log”,  other  excursion  documentation  analyzed  included  digital  audio  recordings  and  extensive  notes 
taken  during  planning  meetings,  interviews,  and  debrief  sessions.  When  possible,  digital  imagery  was 
used  to  also  characterize  the  excursion.  A  hand-held  GPS  receiver  was  used  to  track  the  traversed 
path  and  indicate  waypoints  for  two  of  the  observed  excursions,  with  limited  success. 

For  each  “EVA  Log”,  the  traverse  leaders  described  the  following  parameters  that  described  their 
excursion:  goals,  total  estimated  time,  participants,  number  of  sites  visited,  type  of  mobility  to  sites, 
estimated  distances  between  sites,  and  expected  environmental  factors  that  could  affect  the 
excursion.  Additionally,  an  inventory  of  supplies  was  recorded  and  the  excursion  planning  session 
was  summarized.  After  the  excursion  was  performed,  the  leaders’  debrief  included  unexpected 
events  that  triggered  re-planning. 

Table  3.4  summarizes  all  the  excursions  that  were  documented.  There  were  varied  excursions, 
involving  as  little  as  two  people  or  as  many  as  ten.  Site  selection  was  typically  based  on  the  scientific 
goals  of  the  excursion.  The  exception  to  these  were  traversals  that  were  akin  to  technology  testing, 
such  those  involving  the  mock  spacesuits.  The  type  of  mobility  used  was  determined  by  the 


1  The  duration  of  the  observational  study  was  only  ten  days. 
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distance  between  camp  and  exploration  sites.  The  debriefs  were  essential  in  understanding  the 
factors  that  triggered  re-planning,  and  how  expectations  differed  from  what  actually  occurred  in  the 
excursion.  While  eight  observed  excursions  may  not  seem  many,  the  variety  of  observations  enables 
an  assessment  of  that  is  rich  in  information,  providing  insight  to  future  human  planetary  exploration. 


Table  3.4  Summary  of  all  HMP  excursions  observed  during  2005  season 


1 

2 

$ 

4 

5 

6 

7 

Goal 

Sample 

collection 

Trinity 

Lake 

Prototype 
space  suit 
test 

Sample 
collection, 
image  survey, 
gravity 
measures 

Prototype 
space  suit 
test,  gravity 
measures 

Reconnais¬ 

sance 

excursion 
for  suit 
testing 

Gully 
survey 
along  coast 
line 

Geological 

observations 

Terrain 

assessment 

Haughton 

Crater 

revisit, 

sample 

collection 

Time 

3  hours 

1  hour 

4-5  hours 

2  hours 

4-5  hours 

6  hours 

3-4  hours 

4  hours 

People 

2, 

geologists 

10,  mixed 

4,  researcher 
+  security 

4,  mixed 

6, 

researchers 
+  security 

9,  mixed 

2,  geologist 

9,  mixed 

Sites 

1 

1 

6 

5 

not 

specified 

5 

1 

1  fixed 

Mobility 

ATV,  walk 

ATV,  walk 

ATV,  walk 

ATV,  walk 

ATV 

ATV 

ATV,  walk 

ATV 

Planning 

Repeat  site, 
preset  road. 
Naive  and 
experienced 
pair. 

Repeat  site, 
preset  road. 
Mix  of 
naive  and 
experienced 
people. 

Planning  of 

sites:  order, 

prioritization. 

Expected 

terrain 

difficulties. 

Unique 

opportunity. 

Repeat  site, 
preset  road. 
Naive  and 
experienced 
pair. 

Areas  of 
interest 
identified. 
One 

researcher 

to  wear 
suit, 

majority 

naive 

people. 

Traversing 
>  20  kms 
in 

unexplored 
area.  Maps 
crucial. 
Many 

naive 

people. 

Large  time 
to 

accomplish 
time,  short 
time  of 
travel. 

Accomplish 
sample 
collection 
that  had 
been 

postponed. 
Other  areas 
of  interest 
identified. 

Post¬ 

notes 

2.5  hours 

Unexpected 
delays  due  to 
terrain. 
Postponed 
one  site. 

Extra 
walking.  6 
hours. 

2.5  hours. 
Equipment 
repairs 
done  along 
way. 

Several 
delays  due 
to  terrain. 
Large 
group 
traveled 
slowly.  >  6 
hours 
excursion. 

Large 
group  of 
naive 

travelers, 

terrain 

delays. 

3.3.3.1  Detailed  Excursion  Example:  Into  Haughton  Crater 

One  HMP  excursion  (#3  in  Table  3.4)  is  described  in  detail  below  in  order  to  illustrate  a  traversal 
and  its  similarities  to  Moon  and  Mars  sorties. 
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This  rare  excursion  into  the  middle  Haughton  Crater  was  the  first  of  the  2005  season,  as  special 
permission  is  required  to  enter  the  crater.  There  were  four  participants  in  this  excursion,  of  which 
three  were  experienced  scientists  that  had  visited  the  area  several  times  in  previous  years.  Each 
researcher  had  a  clear  objective:  1)  taking  gravity  measurements  at  multiple  sites,  2)  conducting  a 
gully  survey,  or  3)  collecting  soil  samples  at  one  particular  site.  These  objectives  are  part  of  a 
longitudinal,  on-going  investigation  of  Haughton  Crater,  and  are  critical  for  the  prolongation  of 
HMP  as  a  research  site.  The  fourth  person  in  the  excursion  was  a  safety  officer  (necessary 
protection  in  case  of  polar  bears  -  a  hazard  exclusive  to  Arctic  exploration). 

A  couple  of  hours  before  the  start  of  the  excursion,  all  participants  gathered  to  finalize  the  plans. 
While  the  traverse  leader  was  in  charge  of  the  meeting,  there  was  open  communication  between  all 
parties  involved.  Each  researcher  outlined  his  objectives  for  the  traverse,  identifying  specific  sites 
they  wished  to  reach  (Table  3.5).  Furthermore,  each  site  was  prioritized.  They  used  maps,  both 
topographic  and  aerial,  to  determine  a  feasible  route.  This  was  accomplished  rather  quickly  as  all 
three  scientists  had  previously  undertaken  similar  routes  in  past  seasons.  Hence,  they  were  also  able 
to  anticipate  a  few  locations  along  the  path  that  would  be  difficult  to  navigate.  The  meeting  ended 
with  safety  considerations,  securing  essential  items,  like  radios  and  ATVs,  in  place. 

The  participants  initially  planned  to  visit  six  sites  using  the  ATVs.  During  the  planning  session,  the 
scientists  delineated  which  sites  were  most  critical  and  necessary,  and  settled  on  a  preliminary  route 
to  follow  (Figure  3.4).  The  traverse  leader  estimated  a  total  of  4  to  5  hours  to  complete  the 
excursion,  based  on  the  number  sites,  estimated  time  to  reach  each  and  time  spent  at  each  site. 
Additionally,  this  excursion  was  also  unusual  in  the  respect  that  there  was  a  real  time-pressure 
element.  Two  of  the  researchers  were  leaving  HMP  that  evening  on  a  flight  that  could  not  be 
delayed.  Most  excursions  at  HMP  did  not  have  this  time  pressure  as  there  was  no  real  cost  involved 
with  returning  to  camp  late. 
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Table  3.5  Summary  of  sites  and  estimated  parameters  for  Haughton  Crater  excursion 


1  Site 

_ [Tasks 

Path  Distance 

Travel  Time 

Reached?  9 

Junction 

Gravity  measure 

7  —  8  km 

20  —  30  min 

V 

Tripod  Hill 

Gully  survey,  Gravity  measure 

7  —  8  km 

20  min 

V 

Old  Base  Camp 

Soil  sample 

4  km 

15  min 

X 

Anomaly  Hill 

Gravity  measure 

7-8  km 

30  min 

V 

Perseverance  Hill 

Gravity  measure 

<  1  km 

5  min 

V 

HMP  Base  Camp 

Gravity  measure 

13-  14  km 

35  min 

V 

Figure  3.4  Haughton  Crater  excursion,  with  planned  path,  sites,  and  deviations  from  route. 

A  detailed  inventory  of  their  supplies  was  recorded.  Their  supplies  fell  under  three  categories,  which 
were  common  to  all  excursions  at  HMP:  excursion,  critical  and  scientific  supplies.  Excursion 
supplies  included  lunch,  while  critical  supplies  were  items  that  were  related  to  safety,  such  radios  and 
repair  tools  for  their  mobility  (i.e.,  ATV)  like  a  tire  pump.  Scientific  equipment  was  carried  that 
directly  mapped  to  the  objectives  of  each  researcher,  and  included  a  gravimeter,  cameras,  hand-held 
GPS  receiver,  and  sampling  tools. 
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After  the  excursion  (Figure  3.5),  the  traverse  leader  discussed  events  that  triggered  re-planning.  Due 
to  the  wet  weather  earlier  that  season,  the  ground  bearing  strength  of  certain  areas  had  decreased, 
making  it  very  difficult  to  drive  over.  Since  this  was  the  first  incursion  into  the  crater  that  season, 
they  had  not  anticipated  the  terrain  conditions  would  be  that  poor.  The  traverse  leader  estimated 
that  they  had  spent  all  together  over  an  hour  extricating  their  ATVs  from  two  separate  locations. 
These  locations  were  relatively  early  in  the  route  and  were  not  the  anticipated  problem  areas.  The 
excursion  participants  were  left  to  re-assess  their  plan  and  schedule. 


Figure  3.5  Haughton  Crater  excursion  images 

The  first  re-planning  event  was  with  Old  Base  Camp,  the  only  site  where  soil  samples  were  to  be 
collected.  It  was  unreachable  with  ATVs  due  to  the  low  ground  bearing  strength.  Walking  was  too 
time-consuming  as  there  was  a  pressing  need  to  return  to  base  for  the  evening  flight.  Unfortunately, 
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the  alternative  route  to  the  site  the  researchers  had  selected  was  infeasible  due  to  a  terrain 
miscalculation.  Thus,  for  this  traversal,  the  researchers  left  out  the  essential  site,  in  hopes  that  a 
future  opportunity  would  arise  to  return  and  collect  the  necessary  samples. 

The  second  re-planning  event  was  at  Anomaly  Hill  site  (Figure  3.4),  where  the  researchers  opted  to 
walk  to  the  site  as  the  terrain  was  too  weak  to  support  ATV  travel.  While  walking  was  time- 
consuming,  it  was  still  feasible  within  the  time  constraints.  Eventually,  the  researchers  all  returned 
to  camp  in  time  for  the  two  departing  scientists  to  catch  their  flight.  The  excursion  (summarized  on 
Figure  3.4)  took  a  total  time  of  about  six  hours. 

3.3.3.2  Implications  of  HMP  Observational  Study 

There  are  many  important  analogies  that  can  be  drawn  from  HMP  excursions.  These  traversals  are 
more  complex  than  Apollo  EVAs,  involving  a  greater  number  of  people,  trekking  farther  and 
through  much  more  difficult  terrain.  On  the  other  hand,  HMP  is  an  established  ground  research 
site,  and  though  remote,  does  not  necessitate  critical  life-support  equipment  such  as  spacesuits1. 
Regardless  of  the  differences  with  the  Apollo  program,  similar  problems  arise,  such  as  leveraging 
different  types  of  mobility  and  incurring  delays  due  to  terrain  uncertainty. 

The  additional  planetary  EVA  variables  gained  from  the  observed  HMP  excursions  are  summarized 
in  Table  3.6.  These  variables  played  a  role  in  determining  paths,  both  at  the  pre-  and  re-planning 
stages.  While  providing  insight  to  longer  duration  missions,  this  study  emphasized  the  terrain  (and 
environment)  as  one  of  the  main  sources  of  uncertainty.  With  respect  to  the  environment,  weather 
played  an  important  role,  particularly  as  to  how  it  affected  the  terrain  itself.  In  turn,  the  terrain 
uncertainty  (e.g.,  actual  ground  bearing  strength)  affected  path  planning  and  re-planning.  Plans  took 
into  account  potential  problem  areas,  while  unanticipated  ones  triggered  re-planning.  Furthermore, 
the  selected  transportation  modes  (ATV  or  rovers)  interact  with  the  terrain  differently,  affecting 
speed  and  power  consumption.  Combining  different  modalities  of  transportation  (e.g.,  walking  vs. 
driving)  and  being  able  to  switch  during  the  traverse  increased  the  likelihood  of  completing  all  the 


1  It  could  be  argued,  though,  that  the  safety  officer,  protecting  against  polar  bears,  is  a  life-critical 
element. 
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mission  goals.  Overall,  the  HMP  observational  study  was  essential  in  the  identification  of  planetary 
EVA  variables  that  would  affect  path  planning  under  circumstances  that  go  beyond  Apollo,  in 
particular,  long  duration  lunar  stays  and  Mars  human  missions. 


Table  3.6  Planetary  EVA  variables  based  on  HMP  observational  study 


Variable  Category 

Variable  Specifics 

Astro-agent 

Effect  of  mixed  and  large  teams;  experience  (particularly  for  re¬ 
planning) 

Goals 

Task-specific  equipment  (e.g.,  mass) 

Terrain 

Landmarks,  waypoints;  emphasis  on  ground  bearing  strength 

Transportation 

Reliability  of;  trade-offs  between  walking  and  driving;  emphasis  on 
interaction  between  transportation  and  terrain  &  environment 

Environment 

Emphasis  on  recent  past  weather  affecting  future  terrain  conditions 

Infrastructure 

Communication  availability  (interaction  with  terrain  &  environment) 

Consumables 

Total  time  available 

Operation  rules 

Drive-back  requirements  (e.g.,  alternative  configuration  of  ATV  and 
explorers),  safety-critical  supplies  (e.g.,  mass),  continuous 
communication  availability,  operationally  required  communication  stop 

EVA  components 

Alternative  plans 

3.4  Future  EVAs:  Beyond  Apollo  Experience 


Based  on  subject-matter  expert  interviews  and  the  NASA  (2005)  “Exploration  Systems  Architecture 
Study”  (ESAS),  there  are  additional  planetary  EVA  variables  that  are  relevant  to  the  planning  and  re¬ 
planning  of  excursions.  The  variables  summarized  (Table  3.7)  pertain  to  the  types  of  missions  that 
have  yet  to  be  accomplished:  long  duration  human  missions  on  the  Moon  and  Mars. 


Table  3.7  Planetary  EVA  variables  based  on  planned  future  human  missions 


Variable  Category 

Variable  Specifics 

Goals 

Scientific  return  (value)  of  task  and  sites,  repeatability 

Terrain 

In-situ  materials,  remote  sensing  information  (e.g.,  morphology, 
chemistry),  new  sensed  data  (increasing  fidelity) 

Transportation 

Other  types  (pressurized  rovers,  campers),  preparation  time 

Environment 

Radiation 

Infrastructure 

Bases,  refueling  stations,  navigation  system  (e.g.,  global,  inertial) 

Consumables 

Life  cycle  of  equipment 

Operation  mles 

Radiation  limits,  day/night  restrictions,  work-hours  limits, 
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The  durations  of  EVAs  will  impose  additional  time  limitations  that  are  driven  by  total  work 
conducted  by  astronauts,  acceptable  radiation  exposure  per  EVA,  and  restrictions  on  night-time 
traversals.  Mission  architecture  will  determine  the  availability  of  other  modes  of  transportation  (e.g., 
pressurized  rovers),  refueling  stations,  and  navigation  systems.  The  longer  the  mission,  the  more 
information  is  acquired,  and  higher  fidelity  models  will  have  to  be  incorporated  into  the  planning  of 
excursions.  Even  though  these  extended-stay  missions  are  only  at  the  conceptual  design  stage, 
important  variables  that  will  affect  planetary  EVA  planning  and  re-planning  can  be  identified  and 
incorporated  into  the  Planetary  EVA  Framework. 

3.5  Planetary  EVA  Framework 

The  proposed  framework  for  planetary  EVAs  was  derived  based  on  the  case  studies  described,  such 
as  the  Apollo  program,  on-going  robotic  planetary  exploration,  and  HMP  Mars-analog  excursions. 
The  variables,  identified  in  detail  within  each  case  study,  were  re-organized  into  input  categories  or 
outputs  (Table  3.8).  The  output  is  an  optimized  (or  satisficing)  EVA  plan  that  meets  the  goals  of  the 
mission  and  constraints.  The  mission  and  operational  constraints  are  based  on  system  boundaries, 
which  are  imposed  by  the  resources  and  operational  procedures.  The  framework  organizes  these 
variables  and  constraints  that  affect  and  determine  the  planning  and  re-planning  of  future  planetary 
sorties. 

The  framework  presented  in  Figure  3.6  is  only  a  graphical  representation  of  the  identified  variables 
that  need  to  be  accounted  for  in  a  complete  decision  support  aid  for  human  planetary  exploration. 
Thus,  the  Planetary  EVA  Framework  proposes  the  information  requirements  for  such  an  aid,  and  is 
not  meant  to  be  a  comprehensive  representation  that  specifies  interactions  (e.g.,  linear  vs.  non-linear 
relationships  between  variables),  nor  temporal  constraints  that  are  mission  or  context  dependent. 
The  use  of  a  Venn  diagram  in  Figure  3.6  is  utilized  to  illustrate  that  the  nested  constraints  are  based 
on  multiple  sources,  and  the  relationships  between  variables  will  be  defined  by  a  priori  models  (e.g., 
exploration  costs  and  terrain). 
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Table  3.8  Categorization  of  planetary  EVA  inputs  and  outputs 


Type 

Group 

Variable  categories 

Mission  Resources 

Astro-agent 

Transportation 

Infrastructure 

Consumables 

Input 

Mssion  Objectives 

Goals:  sites,  tasks 

Safety  Margins 

Operational  rules 

Physical  Environs 

Terrain 

Environment 

Models 

Exploration  costs:  consumable  rates 

Path 

Output 

EVA  Plan 

Schedule 

Other  costs  along  path 
Contingencies 

59 


3.5.1  Inputs 


The  input  categories  of  a  planetary  EVA  are  mission  resources,  mission  objectives,  safety  margins, 
exploration  cost  models,  and  planetary  environs.  Defining  each  of  these  is  necessary  to  develop  a 
complete  decision  support  aid  from  EVA  planning  and  re-planning. 

Every  EVA  is  conducted  within  a  particular  planetary  environ,  in  this  case,  Moon  or  Mars,  and 
includes  terrain  and  environment  models.  With  respect  to  the  terrain,  vital  characteristics  that 
pertain  specifically  to  path  planning  are  elevation,  slopes,  rock  density,  rock  size,  and  ground  bearing 
strength.  Other  information,  for  example,  chemical  composition,  would  be  more  relevant  for  the 
selection  of  scientific  sites.  With  respect  to  the  environment,  important  models  include  lighting 
conditions  (for  Moon)  and  dust  storms  (for  Mars).  The  terrain  plays  an  important  role  in  mission 
planning  and  re-planning  as  it  is  one  of  the  main  sources  of  uncertainty,  as  exemplified  in  the  Apollo 
EVAs,  current  MER  experience,  and  HMP  excursions.  Decreasing  terrain  uncertainty  can  be 
achieved  by  increasing  the  fidelity  of  terrain  maps.  It  is  important  to  consider  the  uncertainty  of  all 
models  because  all  EVA  plans  will  be  based  on  these  assumptions,  and  thus,  any  model  errors  will 
inevitably  introduce  errors  in  the  plans. 

The  mission  objectives,  or  goals,  are  the  sites  and  tasks  that  need  to  be  accomplished  in  a  single 
EVA.  These  goals  drive  the  planning  of  the  EVA.  Location,  associate  tasks,  and  priority  are  key 
characteristics  of  the  mission  objectives.  The  goals  of  an  EVA  are  not  always  fixed,  as  additional 
sites  and  tasks  may  be  incorporated  to  take  into  account  new  discoveries  or  emergencies.  Thus,  it  is 
also  imperative  to  know  associated  time  and  energy  expenditures  for  specific  tasks  during  re¬ 
planning. 

The  mission  objectives  can  only  be  met  if  the  appropriate  mission  resources  are  in  place.  The 
resources  are  the  elements  available  to  carry  out  the  EVA,  including  the  astro-agents,  the 
transportation  modes,  the  infrastructure,  and  the  consumables.  The  term  astro-agent  is  used 
because  future  (and  current)  planetary  exploration  encompasses  both  humans  and  robotic  agents. 
Each  astro-agent  will  have  a  particular  capability  or  expertise  that  might  be  necessary  to  accomplish 
the  EVA  mission.  Potentially,  a  source  of  uncertainty  may  be  the  effect  of  mixed  teams  on 
accomplishing  an  EVA,  which  may  add  unexpected  delays. 
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Transportation  is  a  key  element  in  all  planetary  EVAs,  and  the  advantages  of  different  locomotion 
options  (e.g.,  walking  or  driving)  need  to  be  exploited  to  maximize  mission  success.  Several 
examples  within  the  case  studies  cited  reveal  a  poor  understanding  of  the  relationship  between  the 
transportation  mode  and  terrain,  resulting  in  re-planning  of  excursions.  Therefore,  it  is  important  to 
understand  how  the  mobility  type  interacts  with  terrain  characteristics  (e.g.,  speed  under  different 
terrain  rock  densities). 

EVA  planning  and  re-planning  will  have  to  also  take  into  consideration  the  infrastructure  in  place  on 
the  planetary  surface,  be  it  a  lander,  a  base,  or  even  re-fueling  stations.  The  type  of  communication 
and  navigation  system  available  is  essential  for  any  real-time  exploration  aid.  It  is  important  to 
determine  if  these  systems  interact  with  the  terrain  or  the  environment  in  order  to  model  these 
relationships  in  the  planner. 

The  fourth  mission  resource  is  the  available  consumables.  The  type  and  amounts  of  consumables 
will  be  determined  by  the  selected  astro-agents  for  the  EVA  mission  (e.g.,  oxygen  for  humans  versus 
power  source  for  robots).  Among  the  mission  resources,  consumables  are  most  important  as  it  is 
these  that  drive  mission  constraints,  discussed  subsequently. 

The  relationship  between  EVA  inputs  (mission  objectives  and  resources)  and  the  planetary  environs 
is  defined  by  the  exploration  cost  models.  These  cost  function  models  would  determine  (or  predict) 
traversed  distances,  durations  of  tasks  (be  it  walking,  roving,  or  working),  energy  consumed  (e.g., 
metabolic  rates  or  power  used),  rate  of  consumables  expended,  and  favorable  environmental 
positions.  Many  cost  functions  are  needed,  as  they  are  astro-agent  and/or  transportation  specific, 
yet  few  models  exist.  Reliable  estimates  of  exploration  costs  (in  particular  time  and  energy 
consumed)  are  crucial  for  the  planning  of  complex  EVAs,  as  miscalculations  will  force  in-situ  re¬ 
planning. 

3.5.2  Constraints  and  Outputs 

The  intersection  of  mission  objectives  and  resources  is  designated  as  the  overall  mission  constraints. 
Mission  constraints  are  akin  to  hard  constraints,  absolute  limitations  imposed  on  the  system.  For 
instance,  the  astronaut  cannot  consume  more  oxygen  than  the  fixed  amount  available.  Other 
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limitations  involve  the  type  of  transportation  used  in  the  EVA,  and  relate,  for  example,  to  the 
maximum  speed  of  the  mobility  or  terrain  that  is  physically  traversable. 

A  subset  of  the  mission  constraints  are  the  operational  constraints,  which  are  similar  to  soft 
constraints  as  they  are  delineated  by  established  operational  procedures,  or  safety  margins.  Soft 
constraints  are  ones  that  can  be  violated  but  are  imposed  on  the  EVA  for  precautionary  reasons. 
For  example,  one  type  of  rule  may  be  that  astronauts  should  not  exceed  a  particular  metabolic  rate 
during  the  traversal.  Another  type  of  safety  margin  is  one  where  an  astro-agent  must  always  return 
to  base  with  10%  of  their  consumables  remaining.  Mission  planners  will  have  to  use  the  exploration 
models  to  assess  if  the  EVA  plan  violates  any  of  the  operational  procedures.  Safety  margins  may 
possibly  be  overly  conservative  estimates  (e.g.,  total  radiation  dosages)  due  to  limited  operational 
experience  of  human  planetary  exploration,  resulting  in  safe,  but  inefficient  EVAs. 

Two  types  of  operational  constraints  are  identified:  mobility  and  operational  obstacles.  Mobility 
obstacles  depend  on  the  type  of  transportation  used,  and  is  further  restricted  by  safety  margins.  An 
area  may  be  inaccessible  by  a  rover  because  the  rock  density  in  that  area  is  too  high,  yet  that  same 
location  could  be  explored  on  foot  (at  the  cost  of  additional  time,  for  example).  Incomplete 
knowledge  of  how  different  transportation  modes  interact  with  the  terrain  will  result  in  poor 
calculations  of  obstacles,  and  hence  affect  path  planning. 

Operational  obstacles  are  areas  are  out-of-bounds  due  to  operation  mles.  For  instance,  future  EVAs 
may  not  permit  exploration  of  areas  that  are  out  of  communication  range.  If  using  a  rover,  “walk- 
back”  requirements  limit  the  maximum  traversable  distance  from  base  or  re-fueling  station.  In 
terms  of  re-planning,  it  would  be  necessary  to  classify  when  these  operational  restrictions  can  be 
violated,  i.e.,  in  case  of  an  emergency. 

Every  EVA  mission  must  satisfy  the  mission  goals  and  constraints.  As  a  result,  the  EVA  plan 
outputs  a  path  with  an  associated  schedule  and  other  exploration  costs.  Ideally,  contingency  plans 
should  also  be  outlined  for  rapid  re-planning  (e.g.,  alternative  routes  to  sites  or  back  to  base). 
Predicting  associated  exploration  costs  at  sites  and  along  path  is  imperative  as  it  is  this  information 
that  will  be  compared  to  the  actual  EVA.  Real-time  discrepancies  will  indicate  reasons  for  re¬ 
planning.  Over  time,  comparative  analysis  between  predictive  and  actual  exploration  costs  will  assist 
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in  developing  more  accurate  models  of  the  cost  functions,  terrain,  environment,  and  their 
interactions  with  transportation  modes. 

3.5.3  Application  of  Planetary  EVA  Framework 

The  Planetary  EVA  Framework  is  a  result  of  comprehensive  assessment  of  the  past,  present  and 
future  human-robot  planetary  exploration.  All  the  identified  variables  affect  the  planning  and  re¬ 
planning  of  extra-vehicular  activities.  Mission  resources  and  objectives  determine  the  mission 
constraints.  These  are  further  narrowed  down  by  the  imposed  safety  margins,  resulting  in 
operational  constraints.  Only  within  the  system  boundaries  can  an  EVA  plan  be  defined,  including  a 
path  with  adjoining  schedule  and  other  costs.  This  framework  is  also  broad  enough  that,  with  a  few 
nomenclature  modifications  (e.g.,  astro-agent),  it  could  apply  to  other  domains  that  involve  multiple 
types  of  agents  and  missions  that  require  planning  and  scheduling.  This  includes  unmanned  ground 
or  air  vehicles,  rescue  teams,  and  diving  expeditions.  The  specifications  of  each  input  category  and 
constraints  have  to  describe  though  the  work  domain  for  that  particular  field,  just  as  the  Planetary 
EVA  Framework  addresses  human  traverses  on  extra-terrestrial  surfaces. 

Before  sending  humans  to  explore  Moon  and  Mars,  we  need  to  have  a  clear  understanding  of  the 
relationships  between  inputs  and  constraints  across  all  type  of  planetary  EVA  architectures. 
Acquisition  of  this  knowledge  will  not  only  serve  the  purpose  of  developing  EVA  planning  decision 
support  aids  but  also  modeling  future  planetary  surface  operations  with  implications  for  logistics 
mission  planning.  While  additional  specification  with  respect  to  the  parameters  and  constraints  is 
possible,  the  proposed  Planetary  EVA  Framework  is  the  first  step  in  establishing  a  common  ground 
to  discuss  human-robot  extravehicular  activities.  For  the  purposes  of  this  thesis,  only  a  small  sub¬ 
section  of  the  framework  is  applied  for  the  developed  prototype  EVA  path  planner. 
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4  PATH  Prototype:  A  Decision 

Support  Aid  for  Lunar  &  Planetary 
Path  Planning 


4.1  Developing  a  Path  Planner  Prototype 

In  order  to  investigate  how  humans  conduct  path  planning  optimization  with  computer  support,  a 
prototype  automated  path  planner  was  developed.  A  subsection  of  the  Planetary  EVA  Framework 
(Figure  4.1)  was  selected  to  develop  the  path  planner  aid.  From  the  mission  resources,  the  aid 
considers  traverses  that  are  accomplished  on  foot  by  a  suited  astronaut.  Six  exploration  costs  were 
chosen  with  corresponding  models  that  interacted  with  the  lunar  terrain  model.  A  specific  area  on 
the  lunar  surface  is  used,  and  sites  involve  pre-determined  locations  that  astronauts  must  reach.  In 
terms  of  the  environment,  the  only  variable  considered  is  sun  position.  Operational  constraints 
imposed  were  purely  based  on  the  interaction  between  mobility  type  and  slope  (and  hence,  terrain 
elevation).  The  decision  aid  concerns  itself  with  the  path  plan  output.  While  this  might  be 
considered  a  limited  subset,  the  selected  scenario  resembles  the  early  Apollo  missions,  and 
potentially,  the  first  human  missions  back  to  the  Moon.  In  addition,  the  path  is  a  fundamental 
component  of  any  EVA  sortie  as  it  determines  all  exploration  costs  (like  oxygen  consumed)  and 
establishes  the  schedule.  The  prototype  path  planner  aid  is  the  basis  from  which  a  more  complex 
and  complete  decision  support  aid  for  planetary  exploration  can  be  developed.  It  is  a  generic 
interface  that  could  be  applied  to  other  path  planning  domains  beyond  human  planetary  exploration. 
There  are  only  a  few  inputs  that  are  specific  to  Lunar  exploration,  of  which  all  are  interchangeable, 
thus,  applicable  to  Earth  and  Mars. 
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Figure  4.1  Subset  of  Planetary  EVA  Framework  applied  to  prototype  decision  support  aid 

This  chapter  describes  the  developed  path  planner,  its  features  and  interface  specifics.  The 
prototype,  named  PATH  (Planetary  Aid  for  Traversing  Humans,  Figure  4.2),  was  developed  over 
the  course  of  a  year  and  is  written  in  Java  (version  1.5).  The  path  planner  has  as  inputs  a  terrain 
map,  an  obstacle  map,  cost  function  models,  and  other  environmental  conditions.  There  are  two 
methods  in  which  a  path  can  be  defined:  1)  either  the  user  manually  makes  the  entire  path,  or  2)  the 
automation  generates  portions  of  the  path.  PATH  also  provides  the  users  with  the  capability  of 
always  manually  modifying  paths.  There  are  three  possible  map  visualizations  that  have  been 
developed  and  tested.  Finally,  PATH’s  interface,  which  includes  tables  and  tabs,  was  modified 
between  experiments,  and  hence,  there  two  interfaces  are  described.  All  of  these  PATH  features 
(e.g.,  maps,  cost  functions,  visualizations)  are  explained  in  detail  in  this  chapter. 
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Figure  4.2  Prototype  path  planner,  PATH  (Planetary  Aid  for  Traversing  Humans) 


4.2  Terrain  Maps 

This  section  includes  a  description  of  the  path  planner’s  inputs:  a  terrain  elevation  map  and  an 
obstacle  map.  Each  is  pre-defined  by  the  experimenter.  Within  the  scope  of  this  research,  two 
terrain  maps  were  used,  based  on  lunar  terrain  data.  This  component  of  the  path  planner  is 
interchangeable,  as  the  experiment  can  define  any  elevation  and  obstacle  map. 


4.2.1  Cone  Crater:  Apollo  14 

While  global  mappings  of  the  Moon  are  available,  high  resolution  terrain  maps  of  smaller  areas  are 
scarce.  In  a  previous  study  (Carr,  2001),  Carr  interpolated  a  5-meter  resolution  elevation  map  of  the 
Apollo  14  landing  site  of  Cone  Crater  (about  3.65°  south,  17.47  °  west  on  Moon),  an  area  roughly 
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3.2  km2.  While  there  are  other  explored  areas  on  the  Moon  that  have  more  varied  and  interesting 
terrains,  high  resolution  elevation  maps  either  do  not  exist  or  are  not  accessible.  Furthermore,  this 
Cone  Crater  area  was  a  site  that  has  already  resulted  in  a  problematic  expedition  (Apollo  14).  The 
area  is  marked  by  the  large  crater  on  the  north-east  corner,  including  high  elevations  around  the 
southern-face,  and  low  elevations  on  the  western  portion  of  the  map  (Figure  4.3). 


Figure  4.3  Three-dimensional  map  of  original  Cone  Crater  area,  terrain  used  on  PATH 

This  original  terrain  map,  as  depicted  in  Figure  4.3,  was  used  in  the  first  set  of  experiments. 
Subsequently,  the  terrain  was  deemed  as  not  having  enough  changes  in  slopes  and  limiting  the 
choice  of  trials  for  experimentation.  A  modified  version  of  the  terrain  map  was  generated  (Figure 
4.4).  Based  on  the  original  elevation  maps  of  the  Cone  Crater  area,  additional  small  craters  and 
“mole-hills”  were  inserted  on  the  elevation  maps.  The  changes  not  only  allowed  for  more  diversity 
in  trial  choice,  but  also  increased  the  complexity  of  each  trial  such  that  there  was  not  always  one 
clear  best  path. 
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Figure  4.4  Three-dimensional  map  of  modified  Cone  Crater  area,  terrain  used  on  PATH 

4.2.2  Generating  Obstacles 

The  obstacles  used  in  the  path  planner  are  strictly  based  on  terrain  slope.  Figure  4.5  depicts  these 
obstacles  for  the  original  and  modified  Cone  Crater  terrains.  If  the  slope  at  a  particular  point  was 
too  steep  for  a  suited  astronaut,  it  was  considered  an  obstacle.  A  conservative  slope  angle  was 
selected,  15  degrees,  as  the  maximum  slope  suited  astronauts  could  climb  or  descend.  During  the 
Apollo  program,  spacesuits  were  pressurized  and  bulky,  thus  greatly  limiting  the  astronaut’s  mobility. 
If  a  point  was  surrounded  by  steep  terrain  (such  as  the  bottom  of  a  crater),  this  area  became 
inaccessible,  and  thus,  also  an  obstacle.  While  other  types  of  obstacles  are  possible  (such  as  terrain 
that  is  too  rocky  to  traverse),  PATH  restricts  itself  to  one  type  of  obstacle  based  on  terrain  slope. 


Figure  4.5  Obstacles  (slopes  greater  than  15  degrees)  for  terrain  maps  used  in  PATH.  Left, 
original  Cone  Crater  terrain;  right,  modified  Cone  Crater  terrain 
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4.3 


Cost  Functions  Models 


PATH  also  takes  as  input  a  selected  cost  function  (or  objective  function)  which  determines  a  path’s 
cost.  Every  generated  path  has  an  associated  cost.  The  most  commonly  used  cost  function  within 
path  planners  is  distance.  For  PATH,  any  cost  function  is  permissible,  as  long  as  it  grows 
monotonically  (i.e.,  the  cost  from  one  point  to  another  is  always  greater  than  zero)  and  is 
independent  of  path  history  (i.e.,  the  cost  function  does  not  change  along  the  path).  These 
constraints  are  a  consequence  of  the  type  of  search  algorithms  implemented  in  PATH.  A  total  of  six 
cost  functions  were  tested  in  two  different  experiments:  Elevation  Score,  Sun  Score,  Distance,  Time, 
Metabolic  Cost,  and  Exploration  Cost.  Each  is  subsequently  described  in  the  following  sections. 

4.3.1  Sun  Score  and  Elevation  Score 

Surface  visibility  is  a  key  factor  in  the  success  of  lunar  sorties.  In  particular,  sun  glare  was  an  issue 
during  navigation  (both  on  the  rover  and  on  foot)  and  scientific  observations.  If  the  sun  is  directly 
in  front  of  the  astronaut,  there  is  too  much  glare  to  look  forward  (Figure  4.6).  On  the  other  hand,  if 
the  sun  is  directly  behind  the  astronaut,  there  could  be  exaggerated,  misleading  shadows,  obscuring 
the  real  size  of  slopes  and  obstacles.  A  low  sun  elevation  (angle  above  the  horizon)  will  likely  result 
in  large  shadows  and  the  sun  at  “high  noon”  will  create  no  shadows.  It  is  thus  most  favorable,  for 
perception  and  navigation  purposes,  if  the  sun  does  not  create  glare  but  still  provides  some  shadows 
on  the  terrain. 


Figure  4.6  Sun  glare  when  returning  to  the  lunar  module  (NASA  image:  AS14-67-9367) 
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In  order  to  quantify  favorable  sun  positions,  Carr  created  a  cost  function,  Sun  Score,  that  assessed 
the  sun  position  along  the  path  trajectory  (Carr  et  al.,  2003).  The  Sun  Score1  (SS),  Equation  (1),  is  a 
function  that  takes  into  account  the  relative  azimuth  angle,  0,  and  the  relative  elevation  angle,  (j), 
between  the  sun  and  the  observer  (Figure  4.7).  Equation  (2)  defines  0,  which  is  the  difference 
between  sun’s  azimuth  angle  (0sun)  and  the  observer’s  orientation  (0obsetvet)  relative  to  East  (0 
degrees).  Equation  (3)  defines  (j),  which  is  the  difference  between  the  sun’s  elevation  angle  (cj)sun),  and 
the  terrain  slope  the  observer  is  at  (<j>0bServet)- 


Figure  4.7  Sun  position  angles  (left),  relative  azimuth  angle  (center)  and  relative  elevation 

angle  (right) 

With  respect  to  the  Moon,  the  sun’s  elevation  and  azimuth  angles  are  considered  fixed  over  the 
course  of  one  EVA  (about  8  hours)  as  the  rotation  period  of  the  Moon  is  28  days  (as  opposed  to  24 
hours  like  on  Earth).  The  Elevation  Score,  ES  (4),  is  a  subset  of  the  Sun  Score,  only  taking  into 
account  elevation  angles.  Ultimately,  a  high  Elevation  or  Sun  Score  means  that  the  lighting 
condition  is  not  ideal,  while  low  scores  indicate  better  lighting  conditions  for  traversing.  The  optimal 
sun  angle  is  that  which  causes  the  most  contrast  across  the  surface,  which  is  when  the  sun  is  high  in 
the  sky  (approaching  90  degrees  in  relative  latitude)  and  at  a  cross  angle  from  the  direction  of  travel 
(at  90/270  degrees  in  relative  longitude).  If  required  to  walk  towards  the  sun,  the  best  direction  of 
travel  is  45  degrees  in  relative  longitude. 


1  Carr’s  original  Sun  Score  ranged  from  0  to  1.  However,  the  cost  functions  implemented  have  to 
grow  monotonically,  and  thus,  the  function  was  adjusted  to  take  this  into  account. 
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SS  =  (cos(20)  +  2)  •  (cos(2^)  +  2) 


(1) 


G  =  0„-0. 


observer 


(2) 


$  fis tin  'observer  (3) 

ES  =  c  os(2^)  +  2  (4) 


4.3.2  Distance 

The  cost  function  of  Distance  is  based  on  actual  distances  between  points  on  the  map  and  is 
dependant  on  the  map  resolution.  For  example,  the  terrain  maps  used  had  a  resolution  of  5  m,  thus 
a  lateral  motion  would  be  a  distance  of  5  meters  while  a  diagonal  one,  would  be  about  7.07  meters. 
The  shortest  distance  is  achieved  through  a  straight-line  path,  while  avoiding  obstacles. 

4.3.3  Time  Cost 

The  Time  cost  function  is  equivalent  to  duration  of  the  sortie:  how  long  does  it  take  an  astronaut  to 
get  from  one  location  to  another.  The  Time  cost  function  incorporates  both  distance  and  terrain 
slope  (one  more  variable  than  the  Distance  function). 

While  several  studies  have  predicted,  measured,  and  simulated  lunar  walking  speeds,  unfortunately, 
there  is  very  little  data  that  relates  astronaut  traverse  velocity  with  terrain  slope.  For  example.  Stone 
(1974)  estimated  from  Apollo  11  video  that  astronauts  traversed  at  a  maximum  speed  of  2.5  m/s, 
though  most  of  the  speeds  are  under  1  m/ s.  During  Apollo  17,  an  average  of  0.75  m/ s  was  used  to 
calculate  the  “walk-back”  requirement  (Jones,  1995).  Apollo  15  measured  maximum  speeds  of  0.61 
m/ s,  but  this  was  based  on  walking  short  distances  around  the  lunar  lander  or  at  geological  stops 
(Jones,  1995).  Minetti  predicts  that  the  optimal  walking  speeds  on  Moon  will  be  0.6  m/s  and  the 
walk-to-run-transition  speed  is  0.8  m/ s  (Minetti,  2001). 

Waligora  and  Florrigan  (1975)  have  the  only  published  data  from  the  second  EVA  during  Apollo  14 
that  relate  actual  distance  traveled,  duration,  and  net  elevation  change  on  the  lunar  surface.  The  data 
they  published  is  plotted  in  Figure  4.8.  Astronauts  during  this  EVA  were  had  a  range  of  walking 
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speeds  between  0.4  -  1.6  m/s,  walking  fastest  on  flat  terrain.  The  net  elevation  change  is  over 
hundreds  of  meters  in  distance,  so  these  velocities  are  only  averages.  Even  though  the  data  points 
are  scarce,  it  appears  that  velocities  dropped  faster  over  positive  terrain  slopes  (i.e.,  uphill)  while  over 
negative  terrain  slope  (i.e.,  downhill),  velocities  decrease  at  a  slower  rate. 


Apollo  14  (EVA-2)  Traversing  Velocities  as  a  function  of  Slope 
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Figure  4.8  Apollo  14  EVA-2,  walking  velocities,  from  Waligora  and  Horrigan  (1975)  data  set 

A  velocity  profile  for  the  Time  cost  function  was  developed  that  resembled  the  Apollo  14  velocity 
data  (Figure  4.9).  This  profile  is  defined  by  Equation  (5),  where  v  is  velocity  in  m/ s  and  a  is  slope  in 
degrees.  The  maximum  velocity  is  1.6  m/ s  and  occurs  at  flat  terrain.  Going  downhill  is  faster  than 
going  uphill.  Time,  Equation  (6),  is  calculated  by  multiplying  velocity  by  the  distance  traversed. 

0.095 -a  +  1.95,  if  {-15  <  a  <  -10} 

0.06 -a +  1.6,  if{-10<a<0} 

v  =  <  (5) 

-0.02 -a +  1.6,  if  {0  <  a  <  6} 

-0.039 -a  +  0.634,  if  {6<a<15} 

Time  =  v  •  Distance  (6) 
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Velocity  profile  as  a  function  of  Slope  for  Time  Cost  Function 


Figure  4.9  Velocity  profile  as  a  function  of  slope  for  Time  Cost  Function 


4.3.4  Metabolic  Cost 

The  Metabolic  cost  function  is  a  measure  of  how  much  energy  is  consumed  by  the  astronaut  while 
traversing  on  the  lunar  surface.  The  Metabolic  cost  function  builds  upon  the  preceding  function, 
and  thus,  integrates  distance,  terrain  slope,  and  velocity  information. 

Several  studies  have  predicted  and  simulated  energy  consumption  rates  while  traversing  on  the  lunar 
surface  (Newman,  Alexander,  &  Webbon,  1994;  Stone,  1974;  Wickman  &  Luna,  1996;  Wortz  & 
Prescott,  1966).  These  are  summarized  in  Figure  4.10.  In  order  to  make  comparisons  across  these 
studies,  energy  consumptions  rates  were  transformed  to  Joules/ second,  assuming  the  Wortz  and 
Prescott  study’s  average  weights  and  velocities,  and  if  necessary,  average  oxygen  consumption 
energy.  In  general,  these  estimates  are  not  far  from  the  average  EVA  metabolic  rates  on  the  lunar 
surface  (Figure  4.1 1),  which  was  273  ±  32  J/s  (Waligora  &  Florrigan,  1975).  Unfortunately,  most  of 
studies  do  not  correlate  the  rates  with  terrain.  Studies  by  Stone  (1974)  and  Wickman  and  Luna 
(1996)  provided  metabolic  cost  models,  but  the  latter  does  not  include  the  effect  of  terrain  slope. 
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Only  the  Apollo  14  data  for  EVA  to  Cone  Crater  correlates  metabolic  rates  with  terrain  slope  and 
velocity  (Waligora  &  Horrigan,  1975),  which  can  be  used  to  assess  metabolic  cost  models. 


Metabolic  Rates  for  Moon  Gravity  solely  as  a  function  of  Velocity  (Slope  =  0,  Unsuited) 
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Figure  4.10  Summary  of  lunar  metabolic  rates  as  a  function  of  velocity  (on  flat  terrain) 
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Figure  4.11  Average  EVA  metabolic  rates  of  astronauts  during  Apollo  missions  (data  from 

Waligora  and  Horrigan,  1975) 
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There  were  two  models  considered  for  the  Metabolic  cost  function,  one  by  Stone  (1974)  and  the 
other  by  Santee  et  al.  (2001).  Stone’s  model,  Equation  (7),  is  a  function  of  velocity  (v  in  km/hr)  and 
a,  slope  in  degrees.  This  model  is  based  on  simulated  lunar  traverse  experiments,  i.e.,  a  fitted  curve 
(Figure  4.12,  left).  It  is  interesting  to  note  the  trends  exhibited  by  the  model:  there  are  high 
metabolic  cost  penalties  uphill  but  not  as  much  as  going  downhill. 

Energy  rate  =  115  +  (0.45  + 1.26-  sin  a  +  2.23  -sin2  a)  TOO  -v  (7) 
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Figure  4.12  Lunar  energy  consumption  rates:  left,  Stone  (1974)  model  and  right,  Santee  et  al. 

(2001)  model 

Santee  et  al.’s  (2001)  model,  Equation  (8),  is  based  on  both  experimental  data  and  physical  walking 
models.  The  energy  rate  model  is  broken  into  two  main  components:  energy  to  move  forward. 
Equation  (9),  and  energy  to  move  up-  or  downhill,  Equation  (10).  Both  components  depend  on 
velocity,  v  in  m/ s,  and  the  mass,  m  in  kg,  of  the  person  traversing.  The  energy  to  walk  up-  or 
downhill  is  a  function  of  both  velocity  and  slope,  a  in  degrees.  Even  though  Santee’s  model  was 
not  developed  for  extra-terrestrial  navigation,  the  model  has  a  gravity  factor  (in  m/ s2),  thus,  it  can  be 
used  for  modeling  energy  rates  for  the  Moon  and  Mars.  Figure  4.12  (right)  plots  the  energy  rates 
predicted  for  the  lunar  case  (at  a  fixed  velocity).  It  is  noteworthy  to  highlight  that  it  exhibits  the 
same  trends  as  the  Stone  model  (low  rates  downhill,  high  rates  uphill). 
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Energy  rate  =  Wkvtl+Wslopt  (8) 

Wkvel  =  [(3.28)  •  *  +  71 .1]  ■  [(0.661)  •  v  •  cos(a)  +  0.1 1 5]  (9) 

0,  if  a  =  0 

Wup  =  ?>.5- m- g-v if  a  >  0  (10) 

Wdom,  =  2.4  •  m- g-v-  sin(a)  •  0.3|a|/7'65 ,  if  a  <  0 

In  order  to  determine  which  energy  consumption  rate  model  to  use,  the  models  were  compared  to 
actual  Apollo  14  energy  rates  for  the  Cone  Crater  EVA  (Figure  4.13).  Apollo  14  EVA  velocity  and 
corresponding  slopes  (from  Waligora  &  Horrigan,  1975)  were  used  to  predict  metabolic  rates  using 
the  Stone  (1974)  model  and  the  Santee  (2001)  model.  These  were  plotted  against  actual  energy 
expenditure  rates  from  the  two  astronauts  on  the  EVA.  On  downhill  slopes,  Santee’s  model  is 
conservative  (overestimates  rates)  while  Stone’s  model  underestimates  rates.  On  uphill  slopes, 
Santee’s  model  predicts  higher  rates  than  Stone’s  model,  but  both  seem  to  underestimate  rates.  Carr 
showed  that  these  discrepancies  are  due  to  the  fact  that  astronauts  were  dragging  a  small  science 
cart,  and  other  than  this,  Santee’s  model  was  a  good  estimator  of  walking  lunar  metabolic  rates 
(Carr,  2001).  For  this  reason,  and  because  Santee’s  model  has  a  gravity  factor  and  is  more 
conservative  than  Stone’s,  it  was  used  in  the  PATH  prototype,  in  combination  with  the  Time  Cost 
function  to  form  the  Metabolic  Cost  function  model  for  PATH,  Equation  (1 1).  In  this  model,  there 
are  high  metabolic  cost  penalties  for  going  uphill  as  opposed  to  downhill  (Figure  4.14). 


Metabolic  Cost  =  Energy  Rate  •  Time 


(11) 
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Actual  vs.  Model  Comparisons  of  Apollo  14,  EVA-2,  Metabolic  Rates 


Figure  4.13  Metabolic  rates  for  Apollo  14  EVA-2  compared  to  modeled  rates 


Metabolic  Cost  per  meter  as  a  function  of  Slope 


Slope  (degree) 

Figure  4.14  Metabolic  cost  (per  meter)  as  a  function  of  slope,  using  Time  Cost 
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4.3.5  Exploration  Cost 


The  Exploration  Cost  is  a  cost  function  that  takes  into  account  all  the  preceding  cost  functions, 
integrating  distance,  terrain  slope,  velocity,  and  sun  position  information.  Exploration  Cost  is  the 
combination  of  the  Metabolic  Cost  function  and  the  Sun  Score.  Sun  Score  is  weighted  against  the 
Metabolic  Cost.  A  1:2  ratio  was  selected  to  indicate  that  Metabolic  Cost  is  more  important  than  Sun 
Score  (sun  position)  as  energy  consumed  is  directly  related  to  oxygen  supply  (and  hence,  safety 
critical).  The  weighting  is  also  a  ratio  that  PATH  users  could  intuitively  understand  (i.e.,  Metabolic 
Cost  is  twice  as  important  as  Sun  Score). 

Exploration  Cost  =  (Metabolic  Cost)  •  (1  + 1/2  •  SS)  (12) 

4.3.6  Increasing  Complexity  of  Cost  Functions 

The  set  of  cost  functions  presented  depend  on  each  other  and  grow  in  the  number  of  variables.  For 
instance,  the  Elevation  Score  only  a  function  of  the  relative  elevation  angle,  (j),  and  the  Sun  Score 
builds  on  Elevation  Score  with  the  addition  of  relative  azimuth  angle,  0.  These  two  cost  functions 
are  tested  in  the  first  experiment.  Distance,  Time,  Metabolic,  and  Exploration  functions  have  a 
similar  relationship  (Figure  4.15).  This  set  of  functions  is  tested  in  second  experiment.  The  Time 
cost  function  introduces  the  additional  variable  of  slope.  In  the  Metabolic  function,  velocity 
becomes  a  more  important  factor,  though  it  still  depends  on  the  same  variables  that  determine  Time 
(slope  and  Distance).  The  Exploration  cost  function  captures  all  possible  cost  functions  by 
including  both  Sun  Score  and  Metabolic  Cost.  This  relationship  between  functions  is  important 
because  during  path  planning,  different  cost  functions  represent  an  increase  in  numbers  of  variables 
manipulated,  and  hence  a  growing  task  complexity. 
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Figure  4.15  Flow  and  interdependences  between  cost  functions 

4.4  Automation  Levels 

There  are  two  methods  that  a  PATH  user  can  define  a  path,  given  start  and  goal  locations.  These 
correspond  to  two  automation  levels,  LOA  2  and  4  (see  section  2.2.1).  Currently,  only  one 
automation  level  at  a  time  can  be  used.  LOA  2  represents  the  condition  in  which  computer  aid  is 
used  primarily  as  a  filtering  tool,  and  the  human  does  most  of  the  problem  solving,  while  with  LOA 
4,  the  automation,  after  prompting  by  the  user,  suggests  a  solution,  leaving  the  problem  solving 
mostly  to  the  computer.  These  two  levels  of  automation  reflect  the  human-computer  function 
allocation,  in  other  words,  how  much  of  the  path  is  decided  by  the  automation  or  the  human.  For 
the  purposes  of  this  research,  these  levels  have  been  renamed  passive  and  active  automation  (LOA  2 
and  4,  respectively)  corresponding  with  the  role  automation  plays  within  the  path  planning  task. 

Within  active  automation,  the  user  only  decides  on  an  intermediate  point  between  the  start  and  goal 
locations,  while  the  automation  generates  least-costly  path  segments  (Figure  4.16,  left).  Users  are 
restricted  to  only  placing  their  intermediate  waypoint  within  a  pre-determined  critical  way-area. 
Within  passive  automation,  the  user  takes  a  more  central  role,  deciding  all  the  waypoints  within  a 
path  (Figure  4.16,  right),  while  the  automation  takes  the  more  supportive  role  of  calculating  all  the 
costs.  These  levels  are  described  in  more  detail  subsequently. 


80 


Figure  4.16  Difference  between  active  (left)  and  passive  (right)  automation 

PATH  initially  starts  out  with  the  experimenter  selected  inputs  (e.g.,  terrain  map,  cost  function)  and 
a  start,  goal,  and  critical  way-area.  The  critical  way- area  is  an  area  on  the  map  of  interest  (see  orange 
polygon  outline  in  Figure  4.2  and  dashed  outline  in  Figure  4.16)  and  represents  a  potential  site  that 
an  astronaut  wants  to  explore.  Every  path  must  cross  through  this  way- area  when  using  PATH. 

With  respect  of  the  interface,  the  only  visual  difference  between  passive  and  active  automation  is 
found  on  the  far  right  of  the  interface  (Figure  4.17).  Within  passive  automation,  the  user  must  click 
on  “Create  Path”  button  before  starting  a  path;  this  informs  the  interface  that  the  user  is  going  to 
make  a  series  of  waypoints  that  define  a  path.  The  user  then  selects  waypoints  on  the  map  terrain 
which  are  connected  by  straight  path  segments.  Once  the  goal  is  selected  as  a  waypoint,  the  path  is 
completed.  Within  active  automation,  the  user  must  make  a  waypoint  in  the  critical  way-area  and 
then  click  on  “AutoPath”.  The  automation  then  calculates  and  plots  a  least-costly  path  from  the 
start  to  that  waypoint  and  from  the  waypoint  to  the  goal.  With  the  inclusion  of  the  way-area,  the 
problem  space  remains  relatively  large,  permitting  the  exploration  of  different  automated  paths. 

The  algorithm  used  to  find  least-costly  paths  is  the  numerical  potential  field  method,  NPFM 
(Barraquand,  Langlois,  &  Latombe,  1992;  Rimon  &  Koditschek,  1992).  NPFM  is  applied  to  a 
discrete  map,  i.e.,  a  grid  space.  PATH’S  terrain  map  is  a  discrete  map,  where  every  grid  cell  (which 
are  5  meters  apart)  is  associated  with  a  terrain  elevation.  In  general,  the  path  planning  algorithm  first 
calculates  the  minimum  total  path  cost  from  every  grid  cell  to  a  pre-determined  goal  location.  The 
minimum  cost  at  the  goal  is  zero.  In  order  to  determine  the  least-costly  path  from  a  pre-determined 
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start  location,  a  gradient  descent  search  is  applied  from  the  start  grid  cell,  ending  at  the  global 
minima  -  the  goal  grid  cell.  Further  details,  advantages  and  disadvantages  of  the  algorithm  are 
discussed  in  a  subsequent  section  (section  4.7).  The  least-costly  path  (a  sequence  of  grid  cells  from 
the  start  to  the  goal)  is  smoothed  using  the  midpoint  line  algorithm  (Breseham,  1965),  creating  the 
path  segments.  This  algorithm  approximates  a  straight  line  between  grid  cells. 


Make  Path  Modify  Path 

Make  Path  Modify  Path 

Auto  Path 
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oe 
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Figure  4.17  Active  (left)  and  passive  (right)  create  path  buttons  on  the  PATH  interface 

The  “Done”  button  (Figure  4.17)  indicates  the  user  has  finished  creating  their  path,  and  the  selected 
path  is  submitted  as  the  least-costly  path. 

4.5  Modifying  Paths:  Sensitivity  Analysis  Tools 

Once  a  path  is  defined,  either  through  passive  or  active  automation,  the  user  is  able  to  modify  paths. 
Currently,  only  three  paths  can  be  seen  at  once  in  the  PATTI  interface  (Figure  4.2);  each  is  color 
coded  for  easy  distinction  (blue,  red,  and  green).  While  the  user  is  not  limited  to  the  number  of 
paths  they  can  make,  only  these  colored  paths  can  be  seen  and  manipulated.  In  order  to  create 
multiple  paths,  PATH  gives  the  users  the  capability  of  clearing  entire  paths  (with  “Clear  Path” 
button,  Figure  4.17)  or  modifying  paths  through  the  “Modify  Path”  tab  functionalities  (Figure  4.18). 
Once  a  path  is  selected  (either  by  clicking  on  the  path  itself  on  the  map  or  selecting  the  path’s 
corresponding  radio  buttons,  which  are  on  the  top-right  corner  of  PATH  interface.  Figure  4.18),  the 
user  can  move  waypoints,  add  waypoints  and  delete  waypoints.  In  order  to  put  into  effect  the  cost 
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changes  of  the  modified  path,  the  user  must  click  on  “OK”  under  the  “Modify  Path”  tab.  Users  can 
utilize  the  “Cancel”  button1  to  undo  a  path  modification  (before  pressing  “OK”). 


d  Blue  Path 


C  Red  Path 
O  Green  Path 


Make  Path 


Add  WP 


Modify  Path 

IV 


Delete  WP 


OK 


Cancel 


Figure  4.18  Modify  path  functionalities  and  path  radio  buttons 

Once  the  “Modify  Path”  tab  is  selected,  the  user  can  move  waypoints  (by  clicking  and  dragging 
waypoints  on  the  map  terrain).  The  “Add  WP”  and  “Delete  WP”  buttons  are  to  add  and  delete  path 
waypoints;  they  are  toggle  buttons,  and  pressing  them  activates  and  de-activates  these  functionalities. 
Users  can  only  add  new  waypoints  on  the  path.  Once  these  are  added,  they  can  be  moved  and 
deleted  as  well.  Only  if  the  modified  path  is  obstacle-free  will  users  be  able  to  update  the  path’s 
cost.  A  warning  is  given  to  the  user  that  he/ she  needs  to  keep  modifying  the  path  until  it  is 
obstacle-free. 

In  summary,  there  are  two  basic  modes  the  user  can  be  in  when  modifying  paths:  moving  or 
adding/deleting  waypoints.  One  can  distinguish  which  one  they  are  in  by  either  observing  if  the 
“Add”  or  “Delete”  buttons  are  depressed  or  by  noting  the  cursor  type.  When  moving  waypoints, 
the  cursor  is  a  cross-hair;  when  adding  or  deleting,  the  cursor  is  a  pointed  hand. 


1  Users  cannot  revert  to  a  previous  path  after  “OK”  is  pressed. 
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4.6 


PATH  Display  Features 


4.6.1  Path  Cost  Information 

Once  a  path  is  made  or  modified,  path  information  is  displayed  in  the  bottom  section  of  the  PATH 
interface.  The  basic  information  provided  includes  the  path  cost  of  each  individual  path,  the  paths’ 
costs  along  waypoints  (cost  profile),  and  the  paths’  elevation  profile.  There  are  two  versions  of  the 
path  information  displayed,  each  corresponding  to  the  two  experiments  conducted  using  PATH. 
The  difference  between  the  first  and  the  latter  is  the  inclusion  of  cost  bar  graphs. 

The  first  version  of  path  information  display  includes  a  table  with  total  path  costs,  a  path  cost  profile 
graph,  and  a  path  elevation  profile  (Figure  4.19,  where  two  paths  are  shown).  The  table  at  the  top 
has  three  columns,  one  for  each  color-coded  path;  there  are  two  rows,  one  for  the  total  Distance  of 
the  path  and  the  other  for  the  cost  based  on  the  cost  function  (in  the  case  of  Figure  4.19,  Sun 
Score).  Below  the  table,  a  cost  profile  is  plotted.  The  color  matches  the  corresponding  path  color. 
This  plot  provides  information  as  to  how  the  path  cost  is  changing  along  course  of  the  path  (x-axis 
is  always  Distance,  while  the  y-axis  is  the  particular  cost  function).  The  end  points  of  the  path  cost 
profiles  are  the  same  total  costs  represented  in  the  table.  Finally,  below  the  path  cost  profile  is  the 
path  elevation  profile,  which  plots  the  elevations  the  path  encounters.  Again,  colors  match  the 
corresponding  path.  If  no  path  exists,  no  information  is  displayed,  as  in  the  case  of  the  “green”  path 
in  Figure  4.19. 


Figure  4.19  Path  information  display,  version  1 
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For  the  second  version  of  path  information  display,  a  cost  bar  graph  was  included  and  color  was 
added  to  the  table  of  costs  (Figure  4.20).  The  path  cost  profile  and  the  elevation  profile  is  still 
included  but  is  shifted  to  the  left  to  allow  for  the  table  and  the  cost  bar  graph.  Color  was  included 
in  the  table  in  order  to  match  the  profiles.  A  cost  bar  graph,  which  has  the  same  information  as 
table  (total  path  cost),  was  included  after  users  asked  for  a  quick  way  of  determining  which  path 
created  had  the  minimum  path  cost.  Color  was  again  implemented  for  ease  of  recognition. 


Path  Costs: 


Figure  4.20  Path  information  display,  version  2 

4.6.2  Other  Display  Features 

On  the  left  side  of  the  PATH  interface,  information  about  sun  position,  cost  function,  and  map 
scale  is  displayed  (Figure  4.21).  In  the  first  experiment,  only  two  cost  functions  were  tested.  Their 
descriptions  fit  in  the  “Environment”  tab,  which  also  includes  sun  position  (azimuth  and  elevation 
angles,  both  numerically  and  graphically).  However,  for  the  second  experiment,  a  more  complex  set 
of  cost  functions  were  tested.  Since  the  functions  are  interdependent,  all  cost  functions  had  to  be 
described.  Therefore,  the  second  interface  version  of  PATH  included  a  “Cost  Functions”  tab 
(Figure  4.21).  Finally,  a  map  scale  is  included  for  completeness. 
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In  the  second  version  of  the  PATH  interface,  a  timer  was  shown  on  the  far  right  of  the  display  (see 
Figure  4.2  for  location  of  timer).  This  timer  was  an  indicator  for  the  user  on  how  long  they  had 
been  path  planning1. 
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Metabolic  Cost  =  M  =  (Wu-vt-i  +  WaopiO-t 
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0  =  elevation  difference  =  $sun  •  a 


Figure  4.21  PATH  display  of  sun  position  and  cost  functions 


4.7  Map  Visualizations 

There  were  three  possible  map  visualizations:  elevations  contours,  levels  of  equal  cost  (LOEC) 
visualization,  and  elevation  contours  with  LOEC.  Initially,  these  visualizations  were  optional,  i.e., 
they  could  be  triggered  on/ off  by  the  user.  However,  currently,  these  visualizations  are  fixed  and 
only  one  can  be  seen  at  a  time. 


i 


Timer  was  included  as  part  of  the  second  experimental  protocol. 
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Figure  4.22  Map  visualizations  that  include  elevation  contours.  Left,  grayscale-filled  contour 
map;  right,  contour  lines  overlaid  on  top  of  the  second  visualization 

The  elevation  contours  visualization,  as  its  name  implies,  provides  elevation  information  such  that 
within  each  contour,  the  elevation  is  relatively  the  same  (within  10  meters).  There  are  two  ways  in 
which  elevations  contours  were  shown  in  PATH:  grayscale-filled  contour  map  and  contour  lines 
overlaid  on  top  of  levels  of  equal  cost  visualization  (Figure  4.22).  For  the  grayscale  map,  white  is  the 
highest  elevation  while  dark  gray  is  the  lowest  elevation  (black  remains  obstacles).  The  elevation 
contours  presented  in  the  visualizations  resemble  topographical  maps,  except  there  are  no  raw 
elevation  markings.  This  design  decision  was  chosen  because  it  was  deemed  the  addition  of 
numbers  would  clutter  the  map  and  that  the  detailed  information  is  provided  in  the  path  elevation 
profiles. 

4.7.1  Levels  of  Equal  Cost  Visualization 

The  levels  of  equal  cost  (LOEC)  visualization  (colored  gradient  in  Figure  4.23)  displays  integrated 
information  about  areas  on  the  map  that  have  relatively  equal  cost.  LOEC  is  based  on  the 
calculations  generated  from  the  numerical  potential  field  method  (NPFM),  which  is  the  algorithm 
used  in  the  active  automation  version.  The  NPFM  requires  a  grid  decomposition  of  the  space  (or 
terrain),  which  is  based  on  the  map  resolution.  Each  location  on  the  map  is  a  grid  cell  with  a 
matching  terrain  elevation.  Obstacles  are  also  pre-determined,  and  the  corresponding  grid  cells  are 
labeled  as  obstacles  before  the  algorithm  solves  for  a  minimum  path.  Path  costs  are  calculated  based 
on  a  cost  function.  Each  of  these  elements  (map  resolution,  obstacle  identification,  and  cost 
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function  accuracy)  is  set  in  advance,  and  thus,  the  visualization  representation  is  only  as  good  as  the 
a  priori  knowledge  available. 


Figure  4.23  Levels  of  Equal  Cost  (LOEC)  visualization 

The  LOEC  visualization  colors  are  based  on  the  grid  cell’s  minimum  total  cost  from  that  location  to 
the  goal.  The  grid  cell’s  minimum  total  cost  can  be  calculated  because  the  terrain,  obstacles  and  the 
cost  function  are  known  a  priori  and  pre-determined.  In  order  to  calculate  this  minimum  total  cost 
for  each  grid  cell,  a  goal  location  is  first  selected  and  given  a  zero  cost.  Then  the  path  cost  of 
traversing  from  a  grid  cell  to  the  goal  is  computed.  The  cost  between  grid  cells  is  determined  by  the 
cost  function  (fixed  experimentally).  All  possible  paths  from  a  grid  cell  to  the  goal  are  attempted  but 
only  the  minimum  total  cost  is  saved.  Essentially,  this  process  is  similar  to  implementing  Dijkstra’s 
(1959)  algorithm,  which  is  an  exhaustive  search  of  total  costs  from  every  “node”  (in  this  case,  grid 
cells)  to  a  reference  “node”  (with  NPFM,  the  goal  grid  cell).  With  respect  to  obstacles,  each  is  given 
a  high  penalty  cost,  thus  minimum  costs  associated  with  obstacles  are  always  high. 

As  a  result,  each  grid  cell  has  a  corresponding  minimum  total  cost  that  is  relative  to  the  goal. 
Together,  these  costs  can  be  considered  a  minimum  total  cost  field  (similar  to  a  potential  field) 
where  the  goal  location  is  the  lowest  point  (with  a  zero  cost)  and  obstacles  are  peaks  of  high  cost. 
Since  this  cost  field  is  relative  to  the  goal  (i.e.,  the  minimum  cost  from  any  location  to  the  goal),  the 
visualization  is  a  static  map  as  long  as  the  goal  remains  the  same.  The  minimum  total  cost  field  is 
rendered  in  color,  and  locations  depicted  in  the  same  color  have  the  same  minimum  total  cost  to  the 
goal  -  thus  color  indicates  equal  levels  of  cost.  The  color  gradient  chosen  is  between  yellow  and 


purple,  as  they  are  complementary  colors1,  and  thus,  create  the  most  contrast  when  side-by-side, 
helping  users  discern  between  different  cost  levels. 

In  order  to  further  describe  the  process  of  generating  the  LOEC  visualization,  a  simple  illustration 
of  how  a  total  cost  map  is  calculated  is  seen  in  Figure  4.24.  The  map  is  defined  as  a  grid  space, 
where  the  cells  representing  obstacles  are  seen  in  black.  A  start  and  a  goal  cell  are  pre-selected, 
depicted  in  Figure  4.24  as  “S”  and  “G”.  The  minimum  total  cost  from  a  cell  to  the  goal  is  calculated 
based  on  the  pre-selected  cost  function.  In  this  example,  the  cost  function  is  the  Manhattan 
distance  (i.e.,  no  diagonal  motions  from  cell  to  cell).  All  the  minimum  total  cost  values  relative  to 
the  goal  are  stored  and  these  are  the  costs  that  are  represented  in  the  LOEC  visualization.  For 
instance,  in  Figure  4.24,  all  the  cells  with  minimum  cost  of  “5”  would  share  a  color.  In  order  to 
determine  a  least-costly  path,  one  searches  for  the  next  smallest  minimum  cost  until  the  goal  is 
reached  (i.e.,  a  gradient  search).  In  the  example  presented,  only  one  optimal  path  is  highlighted  even 
though  there  are  many  possible  least-costly  paths  that  lead  to  the  goal.  This  same  minimum  cost 
map  can  be  illustrated  in  3D  (Figure  4.25),  where  the  peak  cells  are  the  obstacles. 
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Figure  4.24  Illustration  of  LOEC  generation.  Top:  (left)  grid  map  with  start  &  goal; 
(middle)  complete  cost  map;  (right)  an  optimal  path 


1  Additionally,  these  colors  were  chosen  because  they  did  not  overlap  with  the  existing  red,  blue,  and 
green  paths. 
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Figure  4.25  3D  illustration  of  LOEC  generation 

The  rationale  behind  developing  the  LOEC  visualization  is  multifold.  First,  as  mentioned,  the 
visualization  is  based  on  the  algorithm  (NPFM)  that  determines  the  least-costly  path  between  a  start 
and  goal  location.  Thus,  this  visualization  illustrates  the  computations  calculated  by  the  path  planner 
in  order  to  ascertain  an  optimal  path.  The  computations  take  into  account  all  the  variables  and  the 
underlying  cost  function,  which  may  be  a  complex  model.  In  turn,  users  can  draw  upon  this  visually 
summarized,  integrated  information  to  help  them  determine  a  path,  regardless  of  the  number  of 
variables  and  the  complexity  of  relationships  between  variables  (which  will  be  discussed 
subsequently  in  detail). 

Second,  the  notion  of  “equal  costs”  is  similar  to  the  elevation  contours,  where  a  contour  line  (or 
area)  represents  relatively  equal  terrain  elevations.  A  contour  map  is  a  common,  2D  graphical 
representation  of  changes  in  terrain  elevations  that  is  familiar  to  most  people,  thus,  PATTI  users 
could  extend  the  same  metaphor  to  minimum  total  costs.  A  color  map  was  preferred  over  contours 
lines  (or  areas)  because  it  was  deemed  that  this  representation  would  provide  more  granularity  (i.e., 
the  user  could  distinguish  more  levels  of  equal  cost  than  with  contour  lines).  An  alternative 
visualization  was  considered  based  on  the  minimum  total  cost  field:  a  cost  gradient  map 
visualization.  Since  the  minimum  cost  from  every  location  is  known,  the  “cost  slope”  (the  gradient 
of  the  cost  at  a  particular  location)  could  be  calculated  and  displayed  (a  2D  spatial  derivative  of  the 
minimum  total  cost  field).  This  gradient  map  of  cost  was  contemplated  as  a  possible  visualization 
but  it  was  considered  to  be  an  uncommon  graphical  representation,  and  likely  unintuitive  and 
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difficult  for  users  to  interpret.  Furthermore,  some  directional  information  could  not  be  captured  in 
a  gradient  map. 

Third,  the  particular  algorithm  implemented,  NPFM,  fundamentally  relies  on  the  idea  of  a  “force 
field  metaphor”  (Frixione,  Vercelli,  &  Zaccaria,  2001)  and  hence,  leverages  direct  perception 
interaction  (Gibson,  1979).  There  are  several  path  planning  methods  that  could  have  been  used  to 
find  least-costly  paths  (e.g.,  applying  a  visibility  or  a  Voronoi  graph  in  conjunction  with  a  search 
algorithm  like  A*).  The  NPFM  was  applied  because  it  combines  cell-decomposition  space  and 
artificial  potential  fields  for  planning  (Khatib,  1986).  The  potential  fields  was  deemed  particularly 
useful  for  human-computer  interaction  since  it  fundamentally  relies  on  the  idea  of  that  path 
solutions  are  “attracted”  by  the  goal  location  and  “repelled”  by  obstacles. 

This  underlying  metaphor  that  is  the  basis  of  the  visualization  provides  human  decision  makers  an 
intuitive,  perceptually-based  solution  for  a  large  problem  space.  Flence,  the  visualization  leverages 
on  direct  perception  interaction,  an  important  display  design  principle  and  of  particular  relevance  in 
decision  aids  for  complex  domains.  According  to  Gibson,  direct  perception  is  a  process  that 
requires  little  inference  and  relies  on  lower  levels  of  cognitive  control.  If  the  human  decision  maker 
utilizes  perception  to  solve  the  problem  (i.e.,  “pushed”  a  complex  problem  solving  task  to  a  lower 
level  of  cognitive  control),  it  could  produce  superior  performance  (Vicente  &  Rasmussen,  1992). 
The  reason  behind  this  is  that  users  may  potentially  develop  a  realistic  mental  model  of  the  system 
and  its  constraints,  resulting  in  more  correct  decisions  that  are  done  quickly.  The  concept  of 
supporting  direct  perception  is  also  present  in  ecological  interface  displays,  which  Rasmussen  (1999) 
ascertains  makes  constraints  visible  to  the  user.  While  the  LOEC  visualization  is  not  an  “ecological 
interface  display”,  it  still  supports  the  Gibsonian  ecological  approach  that  supports  the  principle  of 
direct  perception  (Burns  &  Hajdukiewicz,  2004). 

The  colors  on  the  LOEC  visualization  provide  an  aid  to  the  user  determining  a  path1.  It  is 
important  to  point  out  that  the  LOEC  visualization  alone  does  not  identify  the  least-costly  path  for 


1  Specific  instructions  about  LOEC  visualization  and  its  use  for  participants  are  found  within  each 
experiment  description. 
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the  user  (does  not  show  paths)  but  rather  presents  information  about  minimum  total  cost  field 
changes  across  the  entire  problem  space.  For  the  case  of  path  planning,  the  problem  space  is  the 
terrain  and  environmental  variables.  Ideally,  the  human  decision  maker  should  make  a  path  that 
follows  a  gradient  descent,  guided  by  the  changes  in  color  (from  purple  to  yellow),  from  the  start 
location  to  the  goal  (see  also  Figure  4.25).  This  means  that  every  “step”  in  the  path  should  be 
proportional  to  the  negative  of  the  gradient  at  a  particular  point.  The  “auto-path”  function  (see 
section  4.4)  identifies  the  least-cosdy  path  in  this  manner  (by  following  the  least  gradient  descent). 

For  instance,  the  least  costly  path  from  the  start  to  goal  locations  goes  through  a  way-area  (the 
orange  polygon  in  Figure  4.26),  and  the  human  decision  maker  has  the  choice  of  specifically 
traversing  through  area  “A”  or  “B”.  In  order  to  reach  the  goal,  it  is  inevitable  to  traverse  through 
some  areas  that  are  purple,  like  the  start  in  this  example  is  in  a  slight  purple  zone.  The  human 
decision  maker  should  attempt  to  follow  the  color  gradient  from  purple  to  yellow.  This  is  not  the 
same  as  constantly  following  a  yellow  contour,  as  this  could  result  in  high  path  costs.  The  color  in 
area  “A”  has  more  purple  than  the  color  in  area  “B”.  This  communicates  to  the  human  decision 
maker  that  the  minimum  total  cost  from  area  “A”  to  the  goal  might  be  greater  than  that  in  area  “B”. 
In  order  to  avoid  higher  path  costs,  creating  a  path  that  goes  through  area  “B”  as  opposed  to  “A” 
would  likely  be  more  advantageous  and  less  costly.  The  human  decision  maker  would  use  this 
information  alongside  the  other  displays  (e.g.,  path  cost  table  and  path  elevation  profiles)  in  order  to 
compare  path  solutions.  Thus,  the  LOEC  visualization  leverages  direct  perception,  allowing  path 
planners  to  perceptually  understand  constraints  and  path  costs  by  visually  representing  the  total  cost 
field. 


Figure  4.26  Example  use  of  LOEC  visualization  for  path  planning 

Finally,  the  LOEC  visualization  provides  an  aggregate  view  of  the  minimum  total  cost  field,  which 
was  calculated  using  all  the  variables  pertinent  to  the  cost  function.  This  may  be  particularly 


92 


important  for  non-intuitive  cost  functions,  such  as  those  that  rely  on  sun  position  as  opposed  to 
terrain  information.  Furthermore,  the  LOEC  may  guide  sensitivity  analysis  as  users  will  want  to 
match  path  segments  with  corresponding  LOEC  colors.  It  is  hypothesized  that  the  LOEC 
visualization  would  be  helpful  for  users  in  understanding  the  complete  problem  space  because  it 
integrates  multiple  variables  and  reduces  a  complex  problem  to  one  that  is  more  intuitively  and 
visually  obvious. 

4.8  Output  Files 

For  experimental  purposes,  once  the  user  completes  planning  a  path,  and  submits  a  path  answer 
through  the  “Done”  button,  the  path  planner  outputs  two  text  files.  The  first  text  file  is  a  summary 
of  the  paths  submitted  (and  is  the  only  file  the  first  version  of  PATH  outputs).  The  second  text  file 
is  a  complete  listing  of  all  the  mouse-clicks  the  user  has  executed  (tracking  of  interaction  between 
user  and  interface). 

The  first  basic  text  file  includes  the  time  spent  making  a  path  and  the  time  modifying  the  path  (in 
milliseconds),  and  the  total  path  cost  of  all  paths  present  when  user  ended  path  planning,  including 
the  waypoints  of  these.  The  second  text  file  the  following  information:  trial  number,  time  stamp, 
mouse-click  stamp  code,  path  cost  (if  applicable),  and  path  waypoints  (if  applicable).  If  a  mouse- 
click  is  associated  with  a  change  in  path  cost  (i.e.,  making  and  modifying  paths),  the  path  cost  and 
waypoints  are  saved. 
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5  Experiment  1:  Optimizing  Sun- 


related  Cost  Functions 


5.1  Experimental  Objectives 

The  primary  objective  of  this  experiment  was  to  better  understand  human-automation  path 
planning,  which  included  verification  of  selected  metrics  as  an  appropriate  method  for  quantifying 
human  optimization.  The  results  of  this  experiment  are  aimed  at  addressing  the  advantages  and 
disadvantages  of  higher  amounts  of  data  integration  and  automation,  and  their  impact  on  human 
path  planning  performance.  A  secondary  objective  was  to  evaluate  PATH  as  a  path  planning  tool 
and  to  validate  the  experimental  protocol  for  a  larger  study  on  a  simpler  test  case.  In  addition,  sun- 
related  cost  functions  were  the  central  focus  as  these  were  deemed  to  be  atypical  (i.e.,  people  do  not 
typically  try  to  optimize  sun  position  or  geometrically  dependent  variables).  These  sun-related  cost 
functions  also  had  incremental  number  of  variables  between  functions,  a  similar  attribute  to  the 
other  path-related  cost  functions  to  be  tested  in  the  follow-on  experiment  (e.g.,  distance,  time, 
metabolic  cost). 

5.2  Experimental  Methods 

5.2.1  Equipment:  PATH  Interface 

The  decision  support  interface  used  was  PATH  (Figure  5.1),  with  which  the  participants  were  able 
to  make,  modify  and  submit  the  least-costly  paths  they  had  planned.  The  experiment  was  tun  using 
the  a  four  screen  computer  workstation,  three  of  which  were  utilized:  one  screen  for  the 
introduction  materials,  a  second  for  PATH,  and  a  third  presented  the  situation  awareness  questions 
asked  between  trials  (Figure  5.2).  The  computer  screens  have  a  16-bit  color  resolution  at  1280  x 
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1024  pixels,  and  the  screen-capture  program  Camtasia®  simultaneously  ran  on  the  computer  in  order 
to  obtain  archival  video  of  the  user  interactions  with  PATH.  The  workstation  was  a  Dell  Optiplex 
GX280  with  a  Pentium  4  processor. 


Figure  5.1  PATH  interface  for  experiment  1,  annotated 
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Figure  5.2  Experimental  set-up 


5.2.2  Task 

For  this  experiment,  participants  were  asked  to  complete  the  following  task:  using  the  given 
computer  interface,  make  an  obstacle-free,  least-costly  path  with  at  least  one  waypoint  within  the 
designated  way-area.  The  cost  for  each  path  is  based  on  a  cost  (or  objective)  function,  pre¬ 
determined  by  the  experimenter.  Each  trial  has  a  given  start  and  goal  locations  as  well  as  a 
designated  critical  way-area.  Least-costly  paths  for  each  trial  always  traverse  the  designated  critical 
way-areas.  These  are  on  the  lunar  terrain  map  provided  via  PATH.  Environmental  conditions,  i.e., 
sun  azimuth  and  elevation  angles,  that  were  relevant  to  the  optimized  cost  function  were  presented 
on  the  interface  and  changed  for  each  trial  (but  not  within  trials).  After  planning  a  path,  subjects 
submitted  their  path  solution.  After  each  trial,  participants  were  asked  a  couple  of  situation 
awareness  questions  that  directly  related  to  the  previous  trial  before  continuing  to  the  next. 
Subjects  had  to  complete  four  path  planning  trials. 

5.2.3  Independent  Variables 

Three  independent  variables  were  tested  in  this  experiment:  type  of  visualization  (3  types  depicted  in 
Figure  5.4,  elevation  contours,  levels  of  equal  cost  (LOEC),  and  combination  of  elevation  contours 
and  LOEC),  level  of  automation  (the  2  levels  described  previously,  passive  and  active),  and  cost 
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function  (2  functions,  Elevation  Score,  ES,  and  Sun  Score,  SS).  While  visualization  type  was  a 
between-subject  variable,  cost  functions  and  automation  type  were  within- subject  variables,  resulting 
in  a  2  x  2  x  3  repeated  measures  design  (Figure  5.3). 
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Figure  5.3  Summary  of  experimental  conditions  for  experiment  1 


Figure  5.4  Possible  map  visualizations,  including  start  &  goal  locations  and  designated  way- 
area.  From  left  to  right,  (a)  elevation  contours  visualization,  (b)  levels  of  equal  cost 
visualization,  and  (c)  both  visualizations. 
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The  independent  variable  of  visualization  was  chosen  as  a  between- subject  condition  because  naive 
participants  were  needed  for  the  test  trials1.  Furthermore,  this  experimental  design  resulted  in  a 
reasonable  time  involvement  per  participant  (about  1.5  hours).  Thus,  participants  were  randomly 
placed  into  one  of  three  different  visualization  groups.  The  trials  that  were  tested  in  the  other  two 
independent  variables,  cost  function  and  automation  type,  were  not  randomized  because  a  previous 
pilot  study2  showed  that  counterbalancing  was  not  effective.  It  was  determined  that  participants 
were  going  to  exhibit  a  learning  curve  regardless  of  presentation  order.  Thus,  the  order  of  easiest  to 
hardest  was  implemented  in  an  attempt  to  equalize  this  learning  effect.  Additionally,  from  the  pilot 
study,  it  appeared  that  using  the  active  automation  first  would  bias  the  passive  path  planning 
strategies.  The  first  two  trials  used  passive  automation,  and  the  last  two,  active  automation.  Within 
each  automation  type,  participants  first  made  an  ES  path  and  then  an  SS  path.  The  selected 
presentation  order  for  tire  cost  function  was  consistent  with  the  increase  in  difficulty  of  the 
functions.  In  summary,  all  participants  were  asked  to  complete  four  possible  path  planning  trials 
(Table  5.1). 

Table  5.1  Summary  of  trial  order  for  experiment  1,  optimizing  sun-related  cost  functions 


Passive  automation  trials 

Active  automation  trials 

Elevation  Score  trial 

Sun  Score  trial 

Elevation  Score  trial 

Sun  Score  trial 

Trial  1 

Trial  2 

Trial  3 

Trial  4 

Participants  were  randomly  distributed  into  one  of  three  possible  groups  which  were  assigned  to  a 
map  visualization:  elevation  contours  visualization,  levels  of  equal  cost  (LOEC)  visualization,  or 
both  visualizations  (Figure  5.4).  This  was  the  only  difference  between  groups;  all  other  elements, 
such  as  tables  and  modifying  path  functionalities,  remained  the  same.  The  elevation  contour  map, 
which  is  the  nominal  map  that  users  would  expect,  directly  presented  elevation  gradients  from  which 
users  could  infer  rates  of  change,  but  no  other  cost  function  information.  With  the  LOEC 
visualization,  participants  were  presented  with  an  aggregate  of  information,  terrain  and  costs.  Since 


1  The  same  trials  could  not  be  shown  between  visualization  group  (if  within-subjects  variables)  as 
this  could  possibly  result  in  a  confounding  learning  effect. 

2  See  Appendix  A  for  a  summary  of  pilot  study. 
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the  LOEC  visualization  was  likely  not  as  familiar  as  the  contour  map,  a  third  visualization  map,  the 
combination  of  LOEC  and  elevation  contours,  was  also  tested.  This  visualization  could  be  powerful 
as  it  provides  the  participant  with  both  the  raw  data  and  the  total  cost  map. 

Two  increasingly  complex  cost  functions  were  used:  Elevation  Score  (ES)  and  Sun  Score  (SS), 
Figure  5.5.  ES  is  considered  a  simpler  function  than  SS  as  it  has  fewer  variables  to  optimize.  ES 
relates  the  sun’s  elevation  angle  to  the  observer’s  elevation  angle,  or  terrain  slope.  SS  not  only 
encompasses  the  same  information  as  the  ES  function,  but  also  includes  the  relationship  between 
sun’s  azimuth  angle  and  the  observer’s  azimuth  angle,  or  direction  of  travel.  The  optimal  sun  angle 
is  when  the  sun  is  high  in  the  sky  (approaching  90  degrees  in  relative  latitude)  and  at  a  cross  angle 
from  the  direction  of  travel  (at  90/270  degrees  in  relative  longitude).  See  4.3.1  for  more  details. 


Figure  5.5  Flow  and  interdependence  between  cost  functions  for  first  experiment 

Participants  were  tested  on  the  two  automation  levels:  passive  and  active  automation.  More 
specifically,  in  passive  automation,  the  user  created  a  path  by  selecting  path  waypoints  that  were 
connected  by  straight  path  segments.  In  active  automation,  the  user  only  defined  one  waypoint,  in 
the  critical  way-area,  and  then  the  automation  calculated  and  plotted  least-costly  paths  from  the  start 
to  the  waypoint  and  from  the  waypoint  to  the  goal. 

5.2.4  Dependent  Variables 

Path  planning  performance  was  measured  by  final  path  cost,  total  time  to  complete  trial  (both  time 
spent  making  and  modifying  path),  and  the  number  of  situation  awareness  (SA)  questions  that  were 
answered  correctly.  Multiple  choice  questions  were  used  as  a  global  measure  of  the  participants  SA, 
which  was  expected  to  vary  depending  on  automation  levels.  After  every  trial,  participants  were 
asked  two  questions  about  the  previous  trial.  SA  is  (a)  the  perception  of  elements  in  the  current 
environment,  (b)  the  integration  and  comprehension  of  these  elements,  and  (c)  the  projection  of 
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future  status  based  on  comprehension  (Endsley,  1995),  and  the  SA  questions  addressed  the  first  two 
characteristics  (Endsley,  1988).  Specifically,  participants  were  asked  about  the  elements  in  the 
display  (e.g.,  sun  positions),  and  the  cost  functions  (e.g.,  how  path  costs  would  be  affected  by 
changes  in  variables).  There  were  a  total  of  eight  multiple-choice  questions  asked,  four  per 
automation  level.  SA  questions  can  be  found  in  Appendix  A. 

5.2.5  Experimental  Hypotheses 

The  first  experiment  with  PATH  was  designed  to  investigate  how  well  participants  performed  in 
creating  least-costly  paths  when  interacting  with  the  passive  and  active  automation  levels,  as  well  as 
the  different  types  of  visualization,  with  increasingly  difficult  cost  functions.  It  was  hypothesized 
that  with  active  automation,  participants  would  be  able  to  create  near-optimal  least-costly  paths, 
regardless  of  the  cost  function,  in  a  shorter  period  of  time  as  compared  to  passive  automation. 
However,  it  was  hypothesized  smaller  errors  and  shorter  times  would  be  achieved  at  the  expense  of 
decreased  situational  awareness.  Finally,  it  was  hypothesized  that  within  the  passive  automation 
level,  the  LOEC  visualization  would  assist  participants  in  creating  least-costly  paths  and  in  shorter 
periods  of  time  as  compared  to  participants  that  did  not  have  that  visualization,  especially  for  more 
difficult  cost  functions. 

5.2.6  Subject  Instructions 

After  participants  were  assigned  to  one  of  three  possible  visualization  groups,  they  completed  a  pre¬ 
questionnaire  (see  Appendix  A)  that  asked  about  their  average  video  game  usage  and  their  self-rating 
on  map  use  experience.  No  color-blind  participants  completed  the  experiment. 

There  were  two  instructional  phases  in  this  experiment.  In  the  first,  participants  were  given  an 
overview  of  the  task  and  their  designated  visualization.  The  written  explanation  for  each  of  the 
three  possible  visualizations  can  be  found  in  Appendix  A.  They  practiced  the  passive  automation 
with  two  trials,  one  for  each  of  the  cost  functions  under  investigation.  There  was  one  practice  SA 
question  given.  After  the  training  trials,  participants  completed  the  test  trials;  they  had  a  five  minute 
time  limit  on  each  trial.  In  the  second  phase,  participants  were  instructed  on  how  the  active 
automation  worked,  and  two  practice  trials  preceded  the  active  automation  test  trials. 
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As  participants  were  conducting  their  path  planning  task,  a  video  capture  of  the  screen  was  taken. 
At  the  end  of  four  trials,  all  subjects  reviewed  their  video  and  described  in  detail  their  cognitive 
strategy  used  in  making  their  optimal  paths.  All  participants  completed  a  post-questionnaire  that 
asked  them  to  rate  the  usefulness  of  elements  within  the  computer  interface. 

5.2.7  Participants 

Twenty-seven  participants  volunteered  for  this  experiment,  with  an  average  age  of  25.7  ±  3.6  years. 
Participants  were  primarily  graduate  students,  with  18  men  and  9  women,  equally  distributed 
between  the  three  visualization  groups.  There  was  no  significant  difference  in  distribution  between 
average  video-game  usage  and  their  self-ratings  on  map  use  experience. 

5.3  Results 

Analyses  were  conducted  on  tire  path  costs  percent  errors,  total  time  to  task  completion,  percent 
time  spent  modifying  path,  and  number  of  correctly  answered  situational  awareness  questions.  An 
alpha  level  of  0.05  for  all  statistical  tests  was  used.  Analyses  of  variance  were  applied  to  analyze  the 
data;  if  the  assumptions  for  these  tests  were  not  met,  non-parametric  tests  were  used  (Kruskal-Wallis 
test  for  between-subject  variables  and  Wilcoxon  Signed  Rank  test  for  within-subject  variables). 

5.3.1  Path  Cost  Errors 

Path  cost  percent  errors  were  calculated  by  comparing  the  path  cost  generated  by  the  participant  to 
the  automation’s  minimum  path  cost.  The  range  of  these  errors  for  each  condition,  visualization, 
automation  type,  and  cost  function  is  depicted  in  Figure  5.6.  Using  non-parametric  tests,  significant 
differences  across  automation  type  and  cost  function  were  found. 

In  general,  participants  were  able  to  optimize  paths  to  within  35%  of  the  theoretical  optimal  least 
cost  even  when  the  cost  functions  were  unfamiliar  and  had  little  automated  decision  support.  When 
provided  with  automated  decision  support,  participants  were  able  to  generate  a  path  to  within  2%  of 
the  optimal.  For  the  ES  cost  function,  a  Wilcoxon  Sign  Rank  test  yielded  a  significant  difference 
across  automation  type  (Z  =  -4.54,  df  =  1  ,p<  0.001),  resulting  in  smaller  path  cost  percent  errors 
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for  active  automation  (M  =  0.24%,  SD  —  0.26)  as  compared  to  passive  automation  (M  =  4.88%,  SD 
—  0.96).  Similarly  for  SS  paths,  the  more  difficult  cost  function,  a  Wilcoxon  Sign  Rank  test  (Z  -  - 
4.54,  df  =  1  ,p<  0.001)  indicated  that  smaller  errors  were  achieved  when  participants  used  active 
automation  (M  —  0.25%,  SD  =  0.17)  as  opposed  to  passive  automation  (M  —  15.53%,  SD  =  6.30). 
This  effect  was  consistent  across  each  of  the  visualizations. 


Figure  5.6  Box-plot  of  path  cost  errors  between  visualizations,  automation  levels,  and  cost 

functions 

Differences  across  cost  functions  were  investigated  within  each  automation  type.  Within  passive 
automation,  a  Wilcoxon  Sign  Rank  test  (Z  =  -4.52,  df  =  \,p  <  0.001)  showed  that  SS  path  errors  (M 
=  15.53%,  SD  =  6.30)  were  significantly  larger  than  ES  path  errors  (M  =  4.88%,  SD  =  0.96).  This 
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effect  was  consistent  across  each  of  the  visualizations.  Within  active  automation,  there  was  no 
overall  difference  between  the  ES  and  SS  path  errors  (Wilcoxon  Sign  Rank  test,  Z  =  -0.77,  df  -  1  ,p 
=  0.440).  This  effect  was  consistent  for  two  of  the  three  visualizations  (elevation  contours 
visualization  and  LOEC  visualization).  For  the  participants  that  had  both  visualizations  (elevation 
contours  and  LOEC),  there  was  a  marginally  significant  difference  across  cost  functions  (Wilcoxon 
Sign  Rank  test,  Z  =  -1.85,  df  =  1  ,p=  0.06):  SS  active  path  errors  (M  =  0.29%,  SD  =  0.17)  were 
larger  than  ES  active  path  errors  (M  =  0.16%,  SD  =  0.06). 

A  Kruskal- Wallis  test  was  performed  on  each  combination  of  cost  function  and  automation  type  to 
examine  the  differences  across  visualizations.  For  each  of  the  four  conditions  (i.e.,  trials),  there  was 
no  significant  difference  between  the  visualizations  (ES  passive,  %2  (2,  N  =27)  =  1.87,  p  =  0.39;  SS 
passive,  %2  (2,  N  =27)  =  0.42, p  =  0.81  ;  ES  active,  %2  (2,  N  =27)  =  3.19 ,p  =  0.20;  SS  active,  %2  (2,  N 
=27)  =  1.27, p  =  0.53). 

Possible  effects  of  gender,  video  usage,  or  map  use  experience  were  investigated.  There  was  no 
effect  of  video  usage  and  map  use  experience.  When  comparing  if  path  cost  errors  differed  between 
men  and  women  participants,  since  the  ratio  was  uneven  (in  this  experiment  it  was  1:2),  gender  was 
not  considered. 

In  summary,  with  respect  to  path  cost  percent  error,  the  visualization  had  no  main  effect  on  the 
subjects’  performance  within  ES  or  SS  regardless  of  level  of  automation.  Participants,  in  general, 
had  smaller  errors  for  ES  paths  than  SS  paths  within  passive  automation.  Overall,  path  cost  errors 
were  smallest  within  active  automation,  where  there  was  no  difference  between  the  cost  functions. 

5.3.2  Total  Time  to  Task  Completion 

Figure  5.7  shows  a  box-plot  of  times  to  task  completion  (in  seconds)  across  visualizations, 
automation  type,  and  cost  function.  A  repeated  measures  analysis  of  variance  (2x2x3)  was 
performed  to  investigate  the  differences  in  total  time  to  task  completion  between  all  the  conditions. 
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Figure  5.7  Box-plot  of  total  time  to  task  completion  (seconds)  across  visualization, 

automation  type,  and  cost  function 


There  was  no  significant  difference  across  visualizations  with  respect  to  total  time  (F(2,24)  =  0.10,^> 
=  0.91).  Expectedly,  participants  took  significantly  less  time  to  complete  the  task  when  using  active 
automation  (M  =  162.44,  SD  =  69.04)  than  when  using  passive  automation  (M  =241.87,  SD  = 
63.77)  (F(l,24)  =  58.10,y>  <  0.001).  This  difference  is  about  1.3  minutes  on  average. 


A  marginally  significant  difference  was  found  across  cost  functions  (F  (1,24)  =  3.70,  p  —  0.07)  for 
completion  time.  On  average,  participants  took  slightly  longer  to  make  ES  paths  (M  =  207.79,  SD  = 
71.82)  than  SS  paths  (M  =  196.52,  SD  =  82.62).  Flowever,  there  was  a  significant  interaction 


105 


between  cost  function  and  visualization  (F(2,24)  =  5.78,  p  —  0.009).  As  seen  in  Figure  5.8,  the 
interaction  was  caused  primarily  by  the  active  automation.  It  appears  that  LOEC  helped  participants 
arrive  more  quickly  at  a  solution  for  the  ES  cost  function.  Participants  that  had  elevation  contours  in 
their  visualization  took  longer  to  make  ES  paths  than  SS  paths,  yet  for  participants  with  the  LOEC 
visualization,  the  opposite  trend  is  seen.  Simple  contrasts  revealed  no  significant  differences  between 
cost  functions  within  specific  visualization  groups. 


Passive  Automation 


Active  Automation 


LOEC 


LOEC 


Visualization 


Visualization 


Figure  5.8  Means  plot  of  total  time  across  visualization  and  cost  functions:  passive 
automation  (left)  and  active  automation  (right) 


In  summary,  when  using  active  automation,  participants  were  able  to  significantly  reduce  the  amount 
of  total  time  spent  completing  the  path  planning  task  by  about  1.3  minutes.  Within  the  passive 
automation,  most  participants  had  the  tendency  to  take  most  of  their  allotted  time  for  the  task.  No 
main  effect  of  visualization  was  detected.  There  was  only  a  marginal  significant  difference  across 
cost  functions,  where  participants  spent  slightly  longer  times  optimizing  ES  paths  than  SS  paths, 
though  there  was  a  significant  interaction  between  visualization  and  cost  functions.  It  appears  that 
LOEC  subjects  spent  more  time  on  SS  paths  than  ES  paths,  while  the  subjects  in  the  other  two 
groups  had  the  opposite  effect. 
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5.3.3  Percent  Time  Spent  Modifying  Path 


Participants,  when  creating  a  least-costly  path  with  passive  automation,  spent  a  large  portion  of  their 
time  modifying  paths  by  moving,  adding,  and  deleting  waypoints.  Assessing  how  much  time 
participants  devoted  to  a  sensitivity  analysis  on  their  path  solution  will  provides  insight  in 
understanding  how  humans  conduct  optimization.  Most  participants  did  not  use  this  functionality 
with  the  active  automation  as  much,  choosing  instead  to  repeatedly  make  new  paths  (see  Figure  5.9). 
There  are  two  possible  reasons  for  this  trend.  First,  participants  may  have  found  modifying  a  path 
was  more  time  consuming  than  just  making  a  new  path  with  the  active  automation;  second, 
participants  likely  thought  they  could  only  make  paths  less  optimal  by  modifying  them  as  the  active 
automation  plotted  least-costly  paths  between  two  waypoints.  In  order  to  further  investigate  this 
trend,  the  analysis  was  restricted  to  just  the  passive  automation,  where  path  modification  was  the 
most  predominate. 

Within  the  passive  automation  conditions,  a  2  x  3  repeated  measures  analysis  of  variance  (Figure 
5.10)  showed  that  there  was  a  significant  difference  between  cost  functions  (F(l,24)  =  7.25,  p  — 
0.01).  Participants  spent  more  of  their  time  modifying  tire  more  difficult  SS  paths  (M  =  50.13%,  SD 
=  25.24)  than  the  ES  paths  (M  =  41.85%,  SD  =  24.25).  There  was  no  main  effect  based  on 
visualization  (F(2,24)  =  1.35,  p  —  0.28))  and  no  interaction  was  found  between  function  and 
visualization  (F(2,24)  =  0.54,  p  —  0.59).  Hence,  participants  proportionally  spent  more  time 
modifying  SS  paths  than  ES  paths  in  the  passive  automation  phase  of  the  experiment,  regardless  of 
which  visualization  was  used. 


107 


a 

o 

■S 

E 

o 

-M 

< 

<u 

’  C/5 
C/5 

PH 


Elevation  Score 


Sun  Score 


Visualization 


Visualization 


a 

o 

■■d 

E 

o 

A-> 

P 

< 


<u 

> 


o 

< 


Visualization 


Visualization 


Figure  5.9  Box-plot  of  percent  time  spent  modifying  path  across  visualization,  automation 

type,  and  cost  function 
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LOEC 

Visualization 

Figure  5.10  Means  plot  of  percent  time  spent  modifying  path  within  passive  automation 

across  visualization  and  cost  function 

5.3.4  Situation  Awareness 

After  each  trial,  two  situational  awareness  (SA)  questions  were  asked,  totaling  four  questions  for 
passive  automation  and  four  for  active  automation  per  participant.  The  distribution  of  correctly 
answered  questions  (regardless  of  visualization)  is  shown  in  Figure  5.11.  Within  active  automation, 
there  was  one  subject  that  answered  no  questions  correctly  and  one  subject  with  all  correct  answers. 
This  distribution  appears  to  be  different  as  compared  to  the  passive  automation  trials  —  where  12 
subjects  answered  all  four  questions  correctly  and  all  subjects  answered  at  least  one  question 
correctly.  A  non-parametric  test  (Wilcoxon  Signed  Rank  test)  was  performed  to  determine  a 
significant  difference  across  automation  type  (Z  =  -3.35,  df  =  1  ,p—  0.001);  participants  answered 
less  questions  correctly  during  the  active  automation  phase  (M  =  2.33,  SD  =  0.78)  than  during  the 
passive  automation  phase  ( M  =  3.19,  SD  —  0.92)  of  the  experiment.  No  difference  in  SA  was  found 
between  visualization  groups  (Kruskal-Wallis  tests,  within  passive  automation  yp  (2,  N  =  27)  =  1.72, 
p  =  0.42;  within  active  automation  yp  (2,  N  =27)  =  0.68, p  —  0.71). 
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Total  SA  Questions  Correct:  Total  SA  Questions  Correct: 

Passive  Automation  Active  Automation 

Figure  5.11  Distribution  of  correctly  answered  situation  awareness  questions  for  passive 
automation  (left)  and  active  automation  (right) 

5.4  Cognitive  Strategies 

Based  on  experimental  results,  observations,  and  participant  debriefs,  two  general  path  planning 
strategies  were  exhibited.  In  one  strategy,  participants  made  one  or  two  paths  and  then  spent  the 
rest  of  their  time  modified  paths,  moving,  adding  and  deleting  waypoints  (manual  sensitivity  analysis 
strategy).  In  the  other  strategy,  participants  made  multiple  paths,  rarely  utilizing  the  modifying  path 
functionalities.  These  strategies  were  split  between  passive  and  active  automation.  During  active 
automation,  all  participants  tended  not  to  manually  modify  their  paths,  choosing  instead  to  make 
multiple  paths  in  order  to  determine  least-costly  paths. 

During  passive  automation,  the  most  frequent  strategy  implemented  was  the  manual  sensitivity 
analysis,  though  some  chose  to  make  multiple  paths.  This  division  was  apparent  by  assessing  the 
distribution  of  average  percent  time  spent  modifying  for  passive  trials  (Figure  5.12),  where  those 
participants  with  an  average  greater  than  40%  applied  the  manual  sensitivity  analysis  strategy.  The 
selection  of  strategy  was  not  based  on  visualization  as  there  was  no  difference  in  distribution 
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(Pearson’s  chi-square  test,  %2(2,27)  =  2.70,  p  =  0.26).  Furthermore,  there  was  no  difference  in 
performance  (based  on  path  cost  errors)  between  the  strategies  (Mann-Whitney  tests,  within  ES,  Z 
=  -0.39,  p  =  0.70;  within  SS,  Z  =  -0.42,  p  =  0.68). 


Average  Percent  Time  Modifying 

Figure  5.12  Histogram  of  average  percent  time  spent  modifying  across  passive  trials 

Participants  who  chose  to  make  multiple  paths  before  selecting  their  path  solution  did  a  type  of 
“whole-path”  sensitivity  analysis.  This  was  most  evident  in  the  active  automation  where  it  was 
observed  that  participants  created  multiple  paths  in  order  to  explore  the  problem  space.  In  this 
automation  architecture,  the  participant  had  to  only  define  one  waypoint  within  the  critical  way-area 
and  the  automation  generated  the  least-costly  path  segments.  Participants  defined  many  different 
waypoints,  examining  how  path  cost  changed  relative  to  where  the  waypoint  was  placed,  essentially 
conducting  a  sensitivity  analysis  within  the  way-area.  Thus,  all  participants  implemented  a  type  of 
sensitivity  analysis,  either  manual  or  “whole-path”,  in  order  to  optimize  their  path  solutions. 

5.5  Discussion 

In  this  experiment,  there  were  three  visualization  groups,  one  with  elevation  contours,  one  with  the 
levels  of  equal  cost  (LOEC),  and  one  with  both  elevation  contours  and  LOEC.  In  order  to  ensure 
that  the  participants  in  each  visualization  group  were  relatively  the  same,  participants  were  randomly 
assigned  to  a  visualization,  balanced  by  gender  and  number  (i.e.,  number  of  participant  per  group). 
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Furthermore,  there  was  no  statistical  evidence  that  there  was  a  group  differences  with  respect  to 
participants’  map-use  experience  or  choice  of  sensitivity  analysis  type.  Thus,  the  participants  within 
the  visualization  groups  were  reasonably  balanced. 

It  was  hypothesized  that  having  the  LOEC  visualization  would  help  participants  make  superior  least- 
costly  paths,  resulting  in  smaller  cost  errors  and  shorter  task  times  than  participants  who  did  not 
have  the  LOEC,  particularly  when  trying  to  optimize  the  path  cost  whose  cost  function  was  more 
complex.  The  preceding  analysis  did  not  demonstrate  a  main  effect  due  to  type  of  visualization  for 
any  dependent  variable  (either  path  cost  error,  total  times,  or  situation  awareness). 

There  are  several  possible  reasons  why  no  effect  based  on  the  type  of  visualization  was  found.  Some 
participants  commented  that  they  had  not  used  the  LOEC  as  a  primary  tool  to  make  their  least- 
costly  path,  citing  reasons  such  as  they  did  not  know  exactly  how  to  use  it.  Another  possibility  is 
that  the  LOEC  visualization  is  not  particularly  useful  for  cost  functions  with  few  variables.  The 
Elevation  Score  (ES)  only  optimizes  one  variable  and  the  Sun  Score  (SS)  contained  two  variables. 
These  functions  might  not  be  complex  enough  to  merit  the  use  of  LOEC.  Finally,  more  training 
time  and  an  increased  participant  pool  could  have  impacted  the  results. 

The  cost  function  factor  produced  a  significant  effect  for  the  three  dependent  variables  which  is 
expected  since  they  represent  increasing  problem  complexity.  For  path  costs  within  passive 
automation,  ES  path  cost  errors  were  significantly  smaller  than  SS  path  cost  errors,  regardless  of 
visualization.  Within  active  automation,  no  overall  difference  between  cost  functions  was  found, 
but  there  was  a  marginally  significant  difference  that  favored  ES  paths  for  the  visualization  that  had 
both  LOEC  and  elevation  contours.  Thus,  active  automation  decreased  the  task  complexity  with 
respect  to  minimizing  path  cost. 

In  terms  of  time  to  completion,  the  cost  function  factor  had  a  marginally  significant  main  effect, 
however,  there  was  a  significant  interaction  between  cost  function  and  visualization.  It  was  expected 
participants  would  spend  more  time  optimizing  the  more  difficult  function,  SS,  regardless  of 
visualization  and  automation  type.  It  appears,  though,  that  with  the  direct  presentation  of  elevation 
(i.e.,  the  visualizations  with  elevation  contours),  subjects  tended  to  tweak  their  solution  for  the 
purely,  elevation-dependent  cost  function  (i.e.,  ES)  more  than  subjects  who  just  had  the  LOEC 
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visualization.  Thus,  the  participants  that  had  the  elevation  contours  in  their  visualization  spent  more 
time  on  the  simpler  ES  function.  This  is  supported  by  the  slightly  lower  percent  time  spent 
modifying  for  LOEC  visualization  group  (compared  to  the  other  2  visualization  groups)  for  the  ES 
cost  function.  These  results  indicate  that  the  visualization  shifts  the  manner  in  which  subject 
chooses  to  conduct  their  sensitivity  analysis.  In  essence,  if  the  information  is  provided  to  the 
subjects,  they  will  use  it,  though  not  necessarily  to  their  advantage.  In  this  case,  the  additional 
information  of  elevation  contours  on  top  of  LOEC  visualization  gave  the  subject  two  factors  to 
tweak  their  solution;  this  introduced  a  slightly  larger  error  margin  between  cost  functions  in  active 
automation  and  subjects  in  this  visualization  group  appeared  to  spend  more  of  their  time  modifying 
paths  than  the  other  two  groups. 

While  the  different  visualizations  did  not  significantly  affect  subject  performance,  the  degree  of 
automation  dramatically  affected  performance  across  a  number  of  dependent  variables.  As 
hypothesized,  when  subjects  used  the  active  automation,  they  made  smaller  path  cost  errors  and 
took  less  time  than  when  using  the  passive  automation,  regardless  of  visualization  and  type  of  cost 
function.  The  smaller  path  cost  errors  is  not  a  surprising  result  as  the  automation  is  assisting  the 
subjects  in  making  the  least-costly  paths  in  a  large,  complex  problem  space.  When  using  active 
automation,  path  cost  errors  were,  at  most,  only  a  few  percentages  over  the  least  possible  cost, 
allowing  subjects  to  drop  about  1.3  minutes  from  solution  times  as  compared  to  passive  automation. 
For  time  critical  tasks,  this  difference  could  be  essential. 

However,  this  improved  performance  in  terms  of  solution  time  came  at  the  cost  of  decreased 
situation  awareness.  In  the  passive  automation  trials,  81%  of  the  subjects  were  able  to  answer  3  or 
more  of  the  4  SA  questions,  while  during  active  automation,  only  41%  of  subjects  performed  equally 
as  well.  This  is  likely  because  subjects  spent  less  time  at  the  task  and  did  not  conduct  the  same  level 
of  sensitivity  analyses  (as  measured  by  the  percent  time  spent  modifying)  as  they  did  during  passive 
automation.  While  the  passive  automation  caused  longer  solution  times,  it  also  ensured  that  subjects 
became  familiar  with  the  problem  and  thus  had  a  better  understanding  of  the  problem  space  than 
did  those  subjects  who  used  the  automation  to  plan  the  routes. 

Moreover,  it  was  clear  that  the  two  different  automation  levels  produced  two  different  planning 
strategies,  which  in  turn  likely  affected  their  situation  awareness.  Participants  did  explore  the 
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problem  space  within  active  automation,  as  supported  by  the  very  small  path  cost  errors,  but  they 
did  so  differently.  They  chose  to  make  new  paths  instead  of  modifying  paths,  thus  eliminating  the 
“manual”  sensitivity  analysis.  This  strategy  saved  them  time,  but  lowered  their  overall  situation 
awareness.  They  paid  less  attention  to  elements  on  the  PATH  interface,  not  comprehending  how 
tire  automation  was  calculating  the  least-costly  path.  When  subjects  had  to  do  most  of  the  problem 
solving  in  the  path  planning  task  (i.e.,  within  passive  automation),  they  appeared  to  conduct  a  more 
thorough  sensitivity  analysis  when  confronted  with  multi-variable  cost  functions  and  both  LOEC 
and  elevation  contours.  Unfortunately,  while  having  the  LOEC  might  have  induced  subjects  to 
apply  more  sensitivity  analysis,  a  strategy  that  might  have  preserved  their  situation  awareness,  the 
ease  and  quickness  of  using  the  active  automation  prevailed.  Situation  awareness  thus  appears  to  be 
a  function  of  both  time  and  sensitivity  analysis  for  the  geospatial  task  of  path  planning. 

5.6  Conclusions  for  Experiment  1 

In  this  experiment,  two  increasing  degrees  of  automation,  three  different  visualizations,  and  the 
interaction  with  optimizing  increasingly  complex  cost  functions  were  examined.  No  main 
visualization  effect  was  found  but  the  effect  of  the  level  of  automation  was  strong  and  consistent 
across  all  dependent  variables.  After  establishing  such  an  effect  in  this  experiment,  the  next 
experiment  addresses  if  different  visualizations  can  affect  performance  within  a  single  automation 
decision  support  architecture. 

When  the  automation  generated  most  of  the  path  for  the  participants,  they  were  able  to  perform 
better  in  terms  of  time  and  path  costs,  but  paid  the  price  of  decreased  situation  awareness.  Having 
the  levels  of  equal  cost  visualization  promoted  sensitivity  analysis,  but  not  within  active  automation, 
thus  it  was  not  enough  to  counter  tire  loss  of  situation  awareness.  While  the  effects  of  automation 
level  are  striking  and  suggest  that  the  best  decision  support  aid  would  be  one  that  is  highly 
automated,  the  situation  awareness  measures  demonstrate  a  significant  drawback  of  highly 
automated  systems  in  the  reduction  of  SA.  Similar  automation  biases,  i.e.,  excessive  reliance  on 
computer-generated  solutions,  have  been  observed  in  other  domains,  such  as  critical  event  diagnosis 
and  time-sensitive  resource  allocation  (Cummings,  2004).  These  results  indicate  that  the  active 
automation  essentially  leveled  the  task  difficulty  across  problem  complexity.  Consequentially 


114 


participants  did  not  look  for  more  information  than  was  necessary.  Such  automation  bias  is 
undesirable  in  time-critical  domains  because  it  increases  likelihood  of  errors  of  omission  and 
commission  (Mosier  &  Skitka,  1996).  Furthermore,  the  decision  maker  is  less  likely  to  perform  well 
under  unexpected  situations  with  decreased  situation  awareness,  preventing  them  from  identifying 
situations  in  which  the  automation  is  not  properly  functioning  and  hindering  them  in  knowledge- 
based  reasoning  that  is  necessary  under  these  circumstances. 
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6  Experiment  2:  Human  Path 


Optimization 


6.1  Experimental  Objectives 

The  objectives  of  this  second  experiment  were  to  further  evaluating  human-automation  path 
planning  performance  with  the  different  visualizations  and  more  complex  conditions.  The  results  of 
the  first  experiment  (Chapter  5)  led  to  focusing  on  only  one  automation  decision  support 
architecture:  passive  automation.  The  active  automation  in  the  first  experiment  led  to  superior 
performance  with  respect  to  path  cost  errors  and  total  time,  but  no  main  effect  due  to  visualization 
was  detected.  Additionally,  in  the  first  experiment  the  cost  functions  were  relatively  simple  (1-2 
variables),  thus  the  visualization  effect  may  be  more  significant  with  more  complex  cost  functions, 
which  will  be  tested  in  this  experiment.  Therefore,  having  established  the  effect  of  active 
automation,  passive  automation  is  only  tested  in  order  to  further  examine  visualization  effect. 

The  first  experiment  also  determined  that  situation  awareness  decreased  with  the  use  of  active 
automation,  which  was  evidence  of  automation  bias.  Situation  awareness  is  most  invaluable  when 
reacting  to  the  unexpected.  Under  these  circumstances,  the  path  planner  needs  to  take  into  account 
the  discrepancies  between  the  expected  and  what  is  actually  happening.  Discrepancies  are  inevitable 
as  path  planning  (before  the  EVA  mission)  will  rely  on  terrain  information  that  may  be  incomplete 
or  have  limited  resolution.  All  predictions  will  be  based  on  this  terrain  information,  and  hence,  map 
resolution  and  accuracy  will  be  a  primary  driver  for  why  underlying  exploration  cost  models  will  be 
incorrect.  For  example,  the  planner  may  predict  the  astronaut  should  have  50%  of  oxygen 
remaining,  but  in  reality,  there  is  only  40%.  Thus  the  astronaut  is  faced  with  using  a  decision  aid 
which  is  based  on  exploration  models  that  are  erroneous  or  incomplete  (typically  called  automation 
“brittleness”).  The  likelihood  of  this  occurring  is  greatest  in  the  early  stages  of  exploration,  when 
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astronauts  are  still  acquiring  higher  fidelity  terrain  and  cost  models.  For  this  reason,  in  this  second 
experiment,  a  degraded  automation  condition  is  tested. 


6.2  Experimental  Methods 

6.2.1  Equipment:  PATH  Interface 

The  decision  support  interface  used  was  PATH  (version  2,  Figure  6.1),  with  which  the  participants 
were  able  to  make,  modify  and  submit  least-costly  paths.  The  experiment  was  conducted  using  the 
Multi-Modal  Watch  Station,  using  the  same  system  set  up  as  in  the  first  experiment  (see  5.2.1),  with 
the  exception  that  only  two  screens  were  used:  one  for  introductory  materials  and  the  second  for 
PATH. 


Figure  6.1  PATH,  version  2,  used  in  Experiment  2 
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The  PATH  interface  version  used  for  this  experiment  included  a  modified  terrain  map  of  Cone 
Crater,  a  timer  clock,  and  user  interaction  tracking,  which  resulted  in  an  additional  output  file  that 
included  every  mouse-click  the  participant  did  during  each  trial  along  with  tracking  the  time  and  cost 
of  every  path  made.  Also,  slight  modifications  on  the  interface  were  made,  such  as  the  bar  graphs 
(lower  right  corner  of  Figure  6.1)  and  the  model  of  the  cost  functions  under  a  new  tab  (left  top 
corner).  The  new  tab  was  added  for  two  reasons:  1)  to  describe  all  the  cost  functions  in  one 
location,  and  2)  to  allow  for  tracking  of  participants’  attempts  to  access  this  information  (as  they  had 
to  click  on  the  tab  to  view  tire  cost  functions). 

6.2.2  Task 

As  in  the  first  experiment,  participants  were  asked  to  make  an  obstacle-free,  least-costly  path  (with  at 
least  one  waypoint  within  the  designated  way-area)  as  fast  as  possible.  The  cost  of  each  path  was 
based  on  a  cost  function,  pre-determined  by  the  experimenter.  Each  trial  had  a  given  start  and  goal 
locations,  a  designated  critical  way-area,  the  sun’s  position  for  that  trial,  and  lunar  terrain  map. 
Least-costly  paths  for  each  trial  always  traversed  the  critical  way-area.  After  planning  and  optimizing 
a  path,  participants  submitted  their  path  solution.  Participants  had  to  complete  a  total  of  6  test 
trials. 

6.2.3  Independent  Variables 

Three  independent  variables  were  tested  in  this  experiment,  but  under  two  experimental  matrices 
(Figure  6.2).  Within  one  matrix,  the  variables  were  type  of  visualization  (3  types  shown  in  Figure 
6.3)  and  cost  function  (4  functions,  discussed  below).  Within  the  second  matrix,  the  variables  were 
type  of  visualization  (3  types),  cost  function  (only  2  selected),  and  type  of  scenario  (nominal  and  off- 
nominal).  The  off-nominal  scenario  represented  a  degraded  automation  condition.  Since  only  a 
subset  of  the  nominal  cost  functions  were  tested  in  the  off-nominal  case,  two  experimental  matrices 
were  required  for  the  analysis  and  statistical  models.  While  the  visualization  type  was  a  between- 
subjects  variable,  the  cost  function  and  scenario  type  were  within- subjects  variables.  Thus,  there  was 
a  3  x  4  repeated  measures  design  for  the  first  experimental  matrix,  and  a  2  x  2  x  3  repeated  measures 
design  for  the  second  matrix  (Figure  6.2). 
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Experimental  Matrix  1 
Visualization  Group 


Experimental  Matrix  2 
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Figure  6.2  Experimental  conditions  and  matrices  for  second  experiment 
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Figure  6.3  Possible  map  visualizations,  including  start  &  goal  locations  and  designated  way- 
area.  From  left  to  right,  (a)  elevation  contours  visualization,  (b)  LOEC  visualization,  and 

(c)  both  visualizations 

As  in  the  first  experiment,  participants  were  randomly  distributed  into  one  of  three  possible 
visualization  groups:  elevation  contours  visualization,  levels  of  equal  cost  (LOEC)  visualization,  or 
both  visualizations  (Figure  6.3).  While  the  visualization  was  different  between  participants,  all  other 
PATH  features  remained  constant,  such  as  the  path  information  displays  and  the  ability  to  modify 
paths. 

The  same  three  visualizations  were  tested  for  similar  reasons  presented  in  the  first  experiment.  As 
all  but  one  cost  function  depended  on  slope  (and  thus,  elevation)  information,  the  elevation 
contours  remained  the  nominal  map  users  would  expect.  The  LOEC  visualization  (aggregate  cost 
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information)  was  automatically  generated  based  on  the  cost  function  models.  The  third 
visualization,  which  presented  both  LOEC  and  elevation  contours,  combined  the  total  cost  map  and 
the  raw  elevation  data. 

Four  increasingly  complex  cost  functions  were  tested  in  this  experiment:  Distance,  Time,  Metabolic, 
and  Exploration  (see  section  4.3  for  detailed  explanations  of  cost  functions).  The  first  cost  function 
was  the  shortest  distance  between  two  points,  i.e.,  Distance  cost  function.  The  Time  cost  function 
depended  on  Distance  and  slope,  as  astronaut  velocity  relied  on  slope.  The  Metabolic  cost  function 
built  upon  the  Time  function,  including  slope  and  velocity,  to  determine  energy  consumed  as  a 
function  of  terrain.  Finally,  the  Exploration  function  combined  Metabolic  costs  and  Sun  Score  as  an 
overall  cost  function.  Sun  Score,  a  quantifiable  measure  of  favorable  lighting  condition,  related  the 
sun’s  position  (azimuth  and  elevation  angles)  with  the  observer’s  direction  of  travel.  This  cost 
function  was  tested  in  the  first  experiment.  Thus,  the  Exploration  cost  function  was  the  most 
complex  function  as  it  captured  the  costs  that  were  fundamentally  related  to  Distance  and  Time  as 
well  as  spatial  information  (relative  sun  position).  See  Figure  6.4  for  summary  of  cost  functions, 
depicting  interdependence  and  increasing  complexity  of  the  tested  functions. 


Figure  6.4  Flow  and  interdependences  between  cost  functions  for  second  experiment 

Participants  were  tested  in  all  four  cost  functions  under  nominal  scenarios  and  only  two  cost 
functions  under  off-nominal  scenarios  (totaling  6  test  trials).  In  the  nominal  scenario,  participants 
were  told  they  could  rely  on  the  PATTI  interface  to  provide  them  with  accurate  path  cost  based  on 
the  four  previously  discussed  cost  functions.  In  the  off-nominal  scenario,  participants  were 
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informed  that  PATH’S  cost  function  models  were  inaccurate.  This  is  the  additional  degraded 
automation  condition.  Under  the  off-nominal  scenario,  participants  were  only  asked  to  optimize 
two  cost  functions:  Time  and  Exploration.  These  two  were  selected  for  the  off-nominal  case  as  they 
represented  a  simple  and  a  complex  function  (two  versus  four  variables). 

Even  though  all  participants  were  asked  to  complete  six  possible  path  planning  trials,  they  only  were 
presented  with  four  sets  of  start  and  goal  locations  (Table  6.1).  In  the  off-nominal  trials,  participants 
repeated  the  start  and  goal  locations  experienced  in  the  Time  and  Exploration  trials,  but  the  maps 
were  rotated  (Figure  6.5)  in  order  to  disguise  the  fact  that  the  same  trial  was  presented  again.  This, 
in  addition  to  not  informing  the  participants  that  the  off-nominal  trials  were  repeated,  was  done  to 
prevent  a  learning  effect  across  scenarios  but  to  also  ensure  a  comparable  level  of  scenario  difficulty. 
As  in  the  first  experiment,  the  order  of  the  cost  functions  was  not  randomized  among  participants 
but  rather  was  presented  in  order  of  increasing  number  of  variables  manipulated  (in  increasing 
difficulty).  Participants  completed  the  nominal  trials  first  and  then  the  off-nominal.  The  rationale 
for  these  experimental  design  choices  is  similar  as  in  the  first  experiment. 


Table  6.1  Summary  of  trial  order  for  experiment  2,  optimizing  path  planning  cost  functions 


Nominal  scenario 

Off-nominal  scenario 

Distance  trial 

Time  trial 

Metabolic  trial 

Exploration  trial 

Time  trial 

Exploration  trial 

Trial  1 

Trial  2 

Trial  3 

Trial  4 

Trial  2  (repeat) 

Trial  4  (repeat) 

Figure  6.5  Terrain  map  example  comparison  between  nominal  (left)  and  off-nominal  (right) 

scenarios  (with  LOEC  visualization) 
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6.2.4  Dependent  Variables 


Path  planning  performance  was  measured  by  final  path  cost,  total  time  to  complete  trial  (both  time 
spent  making  and  modifying  path),  and  path  cost  profiles.  A  path  cost  profile  (example.  Figure  6.6) 
is  the  cost  of  every  path  the  participant  made  over  the  time  they  completed  the  trial.  From  the  path 
cost  profiles,  a  few  other  dependent  measures  were  identified:  tme  time,  differential  cost,  and  non- 
optimal  satisficing  (i.e.,  percent  time  spent  conducting  non-optimizing  satisficing  and  the 
corresponding  cost  surplus). 


Figure  6.6  Path  cost  profile  illustration 

Figure  6.6  illustrates  the  different  measures  generated  from  the  path  cost  profiles.  Point  A  is  the 
cost  for  the  first  path  created,  while  point  C  is  the  submitted  path  cost  with  its  corresponding  total 
time;  point  B  is  the  actual  minimum  path  cost  found  with  its  matching  time.  Tme  time  is  the  time 
in  which  the  participant  actually  found  their  minimum  cost  path  (the  time  at  point  B  in  Figure  6.6). 
Differential  cost  is  the  difference  between  the  submitted  path  cost  and  the  first  path  cost  made  (path 
cost  error  disparity  between  point  A  and  C  in  Figure  6.6).  Finally,  non-optimizing  satisficing  is 
defined  as  any  additional  time  after  a  minimum  path  cost  was  achieved,  where  participants  were 
unsuccessful  in  finding  a  lower  path  cost  solution.  Non-optimizing  satisficing  includes  both  time 
and  cost  errors  between  the  minimum  path  cost  achieved  and  the  submitted  costs.  In  Figure  6.6, 
this  corresponds  to  the  path  cost  error  and  the  time  differences  between  point  B  and  C. 

With  respect  to  total  time  to  complete  task,  it  was  converted  to  time  penalty.  Participants  were 
asked  to  complete  the  task  as  well  and  as  fast  as  possible.  The  time  pressure  was  imposed  by 
showing  the  participant  a  timer,  an  incremental  clock.  The  penalty  time  started  after  4  minutes  (x. 
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tau),  which  was  not  told  to  the  participants.  In  a  pilot  study  with  an  equivalent  task,  when 
participants  were  not  told  to  take  time  into  consideration,  the  75th  percentile  of  total  time  to 
completion  was  ~  4  minutes.  Thus,  this  was  deemed  an  appropriate  time  limit,  as  most  participants 
would  be  able  to  complete  the  task  while  providing  some  time  pressure. 

6.2.5  Experimental  Hypotheses 

This  second  experiment  was  planned  to  examine  how  well  participants  performed  in  creating  least- 
costly  paths  when  interacting  with  PATH,  across  different  visualizations,  within  only  the  passive 
automation  architecture.  Furthermore,  performance  and  strategy  changes  were  of  interest  under 
nominal  and  off-nominal  conditions,  as  well  as  between  the  increasingly  difficult  cost  functions.  It 
was  hypothesized  that  performance  would  deteriorate  as  the  number  of  variables  manipulated  within 
the  cost  functions  increased.  However,  because  the  problem  space  was  made  perceptually  salient, 
participants  with  the  levels  of  equal  cost  visualization  (LOEC)  should  not  experience  the  same  path 
planning  performance  decline  as  those  with  no  multivariate  visualization.  Degraded  performance 
would  be  indicated  by  an  increase  in  path  cost  error,  total  time,  and  true  time.  Furthermore,  it  was 
hypothesized  performance  would  decrease  in  the  off-nominal  scenarios.  However,  participants  with 
the  visualizations  that  contained  elevation  contours  would  be  able  to  optimize  their  paths  better  than 
the  participants  that  just  had  the  levels  of  equal  cost  visualization.  Participants  with  just  the  LOEC 
visualization  would  be  too  dependent  on  their  erroneous  visualization  (a  form  of  automation  bias) 
and  lacked  the  elevation  contours  that  would  assist  them  in  conducting  the  optimization  with 
degraded  cost  models. 

6.2.6  Subject  Instructions 

Participants  were  randomly  assigned  to  one  of  three  possible  visualization  groups.  Participant  from 
the  previous  experiment  were  excluded  from  this  one.  Each  filled  out  a  pre-questionnaire  that  asked 
about  average  video  game  use  and  their  self-rating  on  map  use  experience.  The  rating  (Table  6.2) 
was  changed  from  the  first  experiment  to  be  more  specific  on  map  use  experience.  No  color-blind 
participants  completed  the  experiment.  All  participants  were  also  given  a  “map  planning”  test  which 
is  suggested  to  measure  “speed  in  visually  exploring  a  wide  or  complicated  spatial  field”  (Ekstrom, 
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French,  &  Harman,  1979,  pg.  39).  This  test  was  selected  because  it  required  participants  to  find  tire 
shortest  path  from  one  point  to  another  in  a  map  maze  and  hence,  might  be  a  measure  of  the 
participants’  path  planning  ability. 


Table  6.2  Map  use  experience  self-rating  descriptors 


Ranking 

Map  use  experience  description 

0 

I  have  not  done  much  hiking  and  am  not  familiar  with  using  topographical  maps. 

1 

I  am  familiar  with  topographical  maps,  but  have  not  used  them  to  plan  or  during  hiking  trips. 

2 

I  have  done  hiking  trip  but  I  am  not  very  familiar  with  topographical  maps,  in  particular,  using 
them  during  hiking  trips. 

3 

I  have  planned  and  done  hiking  trips  with  the  use  of  a  topographical  map. 

4 

I  have  planned  and  done  many  hiking  trips  with  the  use  of  a  topographical  map.  I  have  used 
the  map  to  navigate  terrain  beyond  trials  and/ or  to  triangulate  my  position  during  the  excursion. 

There  were  two  instmctional  phases  in  this  experiment.  In  the  first,  participants  were  given  detailed 
instructions  about  tire  task,  the  time  pressure,  the  cost  functions  that  were  to  be  optimized,  and  the 
visualization  they  were  assigned.  They  were  given  a  practice  trial  before  every  nominal  test  trial  for 
each  cost  function,  i.e.,  practice  trial  preceded  every  test  trial.  Participants  were  instructed  that  they 
had  to  minimize  the  path  cost  as  fast  as  possible;  they  were  informed  that  there  was  a  slight 
advantage  in  doing  the  task  in  less  time  but  larger  penalties  for  longer  times. 

If  LOEC  was  part  of  a  participant’s  visualization,  the  cost  map  was  explained.  Because  of  previous 
subjective  feedback  indicating  confusion  over  the  LOEC  visualization  in  the  first  experiment,  a 
more  in-depth  explanation  of  the  LOEC  visualization  was  included  in  the  introduction.  Written 
description  of  each  visualization  can  be  found  in  Appendix  B. 

In  the  second  phase,  participants  were  told  that  within  tire  PATH  planner  “something  is  wrong  with 
the  cost  functions.”  They  were  given  two  graphs  (Figure  6.7)  to  review  and  examine  which  showed 
the  differences  between  the  expected  costs  (under  nominal  scenario)  and  the  observed  costs  (under 
off-nominal  scenario).  The  graphs  were  presented  to  the  participants  so  that  they  could  acquire  a 
notional  idea  as  to  how  the  cost  function  models  were  erroneous.  This  was  designed  to  be 
analogous  to  a  real-world  situation,  where  automation  inaccuracies  were  noticeable  but  could  not  be 
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immediately  addressed.  For  example,  an  astronaut  might  note  that  actual  oxygen  readings  were  not 
what  were  predicted  by  the  path  planner. 
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Figure  6.7  Deviations  of  cost  functions  between  nominal  and  off-nominal  scenarios, 

instruction  comparative  graphs 

A  video  capture  of  the  screen  was  taken  as  participants  interacted  with  the  path  planning  tool.  At 
the  end  of  the  trials,  all  participants  reviewed  their  video  and  discussed  strategies  used  to  optimize 
path.  Finally,  a  post-questionnaire  was  administered  that  asked  participants  to  rate  the  usefulness  of 
elements  of  PATH.  The  same  questions  from  the  first  experiment  were  asked  except  for  an 
additional  usefulness  rating  for  the  new  bar  graphs  and  the  preference  question  (for  only  the 
participants  that  had  both  LOEC  and  elevation  contours  visualizations)  was  modified  to  ask  if 
participants  agreed  with  the  statement:  “I  preferred  the  elevation  contours  over  the  LOEC 
visualization  to  make  least-costly  paths.” 

6.2.7  Participants 

Thirty- four  participants  volunteered  for  this  experiment,  and  were  compensated  ($  10/hour)  for  their 
participation.  Their  average  age  was  25.0  ±  3.5  years.  Participants  were  primarily  graduate  students, 
with  22  men  and  12  women,  equally  distributed  between  the  visualization  groups.  There  was  no 
significant  difference  in  distribution  between  average  video-game  usage,  map  planning  test  score, 
and  their  self-ratings  on  map  use  &  hiking  experience. 
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6.3 


Results 


Analyses  of  variance  were  used  to  test  hypotheses  if  assumptions  were  met.  Otherwise  appropriate 
non-parametric  tests  were  used.  An  alpha  level  of  0.05  for  all  statistical  tests  was  applied.  As  two 
experimental  matrices  were  tested,  the  results  are  divided  into  Phase  1  (3  x  4  nominal  cases)  and 
Phase  2  (2  x  2  x  3  scenario  comparisons  with  the  off-nominal  condition).  No  covariates  were  used 
in  the  subsequent  analysis.  Neither  map  planning  scores  nor  map  &  hiking  experience  were 
significantly  correlated  with  path  cost  errors  and  time  penalty. 

For  detailed  test  results  and  descriptive  statistics,  refer  to  Appendix  B. 

6.3.1  Path  Cost  Errors 

Path  cost  errors  were  calculated  by  normalizing  the  path  cost  generated  by  each  participant  to  the 
automation’s  minimum  path  cost.  Non-parametric  tests  were  used  to  examine  the  differences  across 
conditions. 

6.3.1.1  Phase  1 

The  range  of  path  cost  errors1  for  phase  1  of  the  experiment  are  shown  in  Figure  6.8.  Within  each 
cost  function,  there  was  no  statistical  difference  between  visualization  groups.  As  Figure  6.8 
suggests,  there  was  a  significant  difference  between  cost  functions  (Friedman  test,  p  <  0.001). 
Wilcoxon  Sign  tests  confirmed  the  following  ranking:  the  Time  cost  function  (M  =  1.60,  SD  —  0.14) 
had  significantly  larger  path  cost  errors  (p  <  0.0001),  Distance  had  the  smallest  (M  =  1.03,  SD  = 
0.03),  and  there  was  no  statistical  difference  between  Metabolic  (M  =  1.07,  SD  =  0.02)  and 
Exploration  (M  =  1.06,  SD  —  0.02)  cost  functions.  Within  visualizations,  the  results  were  similar  (in 
agreement  with  the  lack  of  visualization  differences). 


1  Two  data  points  were  identified  as  outliers  (3.44*SD  and  4.28*SD  above  the  mean)  and  were 
removed.  If  data  greater  than  3.29*SD,  as  recommended  by  Tabachnick  and  Fidell  (2001),  it  was 
considered  an  outlier. 
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Figure  6.8  Box-plot  of  path  cost  errors  between  visualizations  and  cost  functions,  Phase  1 


6.3.1.2  Phase  2 


The  range  of  path  cost  errors  for  the  second  phase  of  this  experiment  is  shown  Figure  6.9.  Using 
Kruskal-Wallis  tests,  no  significant  differences  in  path  cost  errors  across  visualizations  were  found 
within  the  Time  nor  the  Exploration  off-nominal  trials.  This  is  consistent  with  the  nominal 
scenario,  which  indicates  that  visualizations  did  not  affect  path  cost  errors.  Similar  to  the  nominal 
scenario,  the  off-nominal  Time  path  cost  errors  (M  —  1.53,  SD  —  0.20)  were  significantly  higher  (p  < 
0.0001)  than  the  off-nominal  Exploration  path  cost  errors  (M  =  1.10,  SD  =0.03). 
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Figure  6.9  Box-plot  of  path  cost  errors  between  visualizations  and  scenario  type,  Phase  2 

Wilcoxon  Sign  tests  were  used  to  compare  across  scenarios  (nominal  vs.  off-nominal  cost  functions). 
Overall,  regardless  of  visualization,  there  was  a  significant  difference  in  path  cost  errors  between 
scenarios.  The  path  cost  errors  for  the  nominal  Time  cost  function  were  significantly  larger  than  the 
off-nominal  errors  (p  =  0.05).  The  opposite  effect  was  found  for  the  Exploration  cost  function, 
where  the  path  cost  errors  were  significantly  lower  in  the  nominal  scenario  (p  <  0.0001). 

Further  comparisons  between  nominal  and  off-nominal  scenarios  were  conducted  within  each 
visualization  group.  For  the  Exploration  cost  function,  participants  within  each  visualization  group 
had  a  significantly  lower  path  cost  errors  for  the  nominal  than  the  off-nominal  scenario.  For  the 
Time  cost  function,  the  difference  across  scenarios  was  driven  by  the  significant  decrease  in  off- 
nominal  path  cost  errors  for  the  LOEC  participants  (p  =  0.026).  No  significant  differences  across 
scenarios  were  detected  for  the  other  two  visualization  groups.  These  results  suggest  that  only  the 
participants  with  the  LOEC  visualization  decreased  their  path  cost  errors  for  the  Time  off-nominal 
trial. 
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In  summary,  within  the  off-nominal  scenario,  visualization  did  not  have  a  main  effect  on  overall 
path  cost  errors,  and  the  Time  cost  function  resulted  in  larger  errors  than  the  Exploration  function, 
which  is  consistent  with  the  nominal  scenario  results.  Across  scenarios,  the  results  were  mixed. 
Within  Time,  errors  decreased  during  the  off-nominal  scenario,  though  it  was  mainly  due  to  the 
performance  of  the  LOEC  participants.  Within  Exploration,  errors  increased  during  the  off- 
nominal  scenario  consistently  across  all  visualization  groups. 

6.3.2  Total  Time:  Time  Penalty 

Total  time  to  complete  the  task  was  transformed  to  time  penalty  (et/T;  t,  total  time,  X,  4  minutes). 
Time  penalty  was  used,  as  opposed  to  total  time,  because  participants  were  instmcted  that  there  was 
a  penalty  for  spending  too  much  time  on  the  task,  i.e.,  they  were  to  optimize  the  path  as  fast  as 
possible.  As  the  time  penalty  is  an  exponential  transformation,  the  median  Levene’s  test  (Brown  & 
Forsythe,  1974)  was  used  to  test  for  equal  variance.  Repeated  measures  analyses  of  variance  were 
performed  to  investigate  differences  between  the  conditions.  If  the  sphericity  assumption1 2  was  not 
met,  the  Greenhouse-Geisser  adjusted  p-value  is  reported. 

6.3.2.1  Phase  1 

A  box-plot  of  the  time  penalties  across  visualizations  and  cost  functions"  is  shown  in  Figure  6.10.  A 
repeated  AN OVA  (3  x  4,  visualization  x  cost  function,  Figure  6.11)  revealed  a  significant  difference 
between  cost  functions  (p  =  0.001)  but  no  significant  difference  across  visualizations  nor  interaction 
between  treatments. 


1  The  sphericity  assumption  is  similar  to  the  ANOVA  requirement  of  homogeneity  of  variances, 
except  that  it  is  within-subjects.  The  assumption  is  that  variances  of  the  differences  (between  pairs) 
across  groups  are  the  same. 

2  One  outlier  (4.14*SD  above  the  mean)  was  removed. 
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Figure  6.10  Box-plot  of  time  penalty  between  visualization  and  cost  functions,  Phase  1 


Figure  6.11  Means  plot  of  time  penalty  across  visualization  and  cost  functions,  Phase  1 
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In  order  to  determine  which  cost  functions  affected  the  participants  the  most,  all  pair-wise 
comparisons  were  conducted  (Bonferroni  test).  The  Time  cost  function  took  the  longest  to 
optimize  as  the  time  penalties  were  significantly  larger  than  the  ones  for  Distance  (p  =  0.003), 
Metabolic  (p  <  0.0001)  and  Exploration  cost  functions  (p  =  0.007).  There  were  no  other  significant 
differences  between  the  other  functions  (i.e.,  equal  time  spent  in  Distance,  Metabolic,  and 
Exploration  trials). 

Even  though  there  was  no  difference  detected  across  visualizations.  Figure  6.11  shows  an  interesting 
trend  for  the  LOEC  visualization  group.  The  two  groups  that  had  elevation  contours  in  the 
visualization  had  an  increase  in  time  penalty  for  the  Time  and  Exploration  cost  functions  only.  On 
the  other  hand,  participants  that  had  just  the  LOEC  visualization  did  not  have  an  increase  for 
Exploration  function,  but  rather  a  slight  decrease.  Additionally,  for  this  visualization  group,  there 
does  not  appear  to  be  much  of  a  difference  across  the  cost  functions  as  there  is  with  the  other  two 
groups.  In  order  to  understand  if  and  how  the  LOEC  visualization  affected  the  time  penalties  (and 
thus  the  total  time  spent  on  the  task),  further  investigation  was  conducted  within  this  visualization. 
A  simple  main  effects  analysis  of  cost  function  within  the  LOEC  visualization  group  showed  that 
there  were  no  significant  differences  between  functions.  The  results  imply  that  the  LOEC 
visualization  participants  spent  relatively  equal  amounts  of  time  optimizing  each  cost  function. 

In  summary,  the  Time  cost  function  took  the  longest  time  to  optimize,  resulting  in  the  highest  time 
penalty.  No  other  significant  differences  between  cost  function  were  detected.  While  no  main 
effect  due  to  visualization  was  detected  using  AN OVA,  the  trend  within  the  LOEC  group  showed 
that  these  participants  spent  relatively  equal  amounts  of  time  for  each  cost  function. 

6.3.2.2  Phase  2 

A  repeated  ANOVA  (2x2x3,  scenario  x  cost  function  x  visualization)  was  implemented  to  detect 
differences  among  the  conditions.  Figure  6.12  shows  the  range  of  time  penalties  for  the  conditions 
tested1.  There  was  only  a  significant  difference  with  respect  to  cost  function  (p  <  0.0001).  There 
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Two  outliers  (4.14*SD  and  3. 5 DSD  above  the  mean)  were  removed. 
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were  no  significant  differences  between  visualization  groups  or  scenarios,  and  no  significant 
interactions  between  the  conditions  were  detected.  This  implies  that  the  scenario  did  not  affect  the 
time  spent  in  trying  to  optimize  paths,  and  only  the  cost  function  caused  a  change  in  total  time 
penalty.  In  this  case,  participants  spent  more  time  optimizing  the  Time  cost  function  than  the 
Exploration  cost  function  (Figure  6.13). 


Figure  6.12  Box-plot  of  time  penalties  between  visualizations  and  scenario  type,  Phase  2 
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Figure  6.13  Means  plot  of  time  penalty  across  visualization,  type  of  scenario,  and  cost 

function  (Time,  left;  Exploration,  right) 


6.3.3  Percent  Time  Spent  Modifying  Path 


Percent  time  spent  modifying  paths  was  calculated  as  the  ratio  between  time  modifying  to  total  time. 
This  is  an  important  measure  because  it  is  an  indicator  of  the  amount  of  sensitivity  analysis 
conducted  during  path  optimization. 


6.3.3.1  Phase  1 

Figure  6.14  shows  the  box-plot  of  the  percent  time  spent  modifying.  A  repeated  measures  ANOVA 
showed  there  was  no  overall  difference  across  functions  or  visualization  group,  and  no  significant 
interaction  between  treatments  was  detected.  While  there  was  no  significant  difference  between 
conditions,  Figure  6.15  suggests  a  trend  based  on  visualization,  which  merits  further  investigation. 
The  trend  appears  to  be  that  when  the  LOEC  was  present  in  the  visualization,  participants  were 
inclined  to  spend  more  of  their  time  modifying  paths  than  participants  who  just  had  the  elevation 
contours. 
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Figure  6.14  Box-plot  of  percent  time  spent  modifying  path  between  visualization  and  cost 

function,  Phase  1 
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Figure  6.15  Means  plot  of  percent  time  spent  modifying  path  between  visualizations  and 

cost  functions,  Phase  1 
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Since  the  trend  appears  to  be  LOEC  participants  were  spending  more  modifying  time,  all 
participants  were  re-grouped  into  two  groups,  those  with  LOEC  and  those  without.  While  no 
significant  main  effect  based  on  LOEC  was  detected,  there  was  a  significant  difference  across 
groups  within  the  Time  cost  function  only  (p  =  0.016).  This  implies  that  for  the  Time  function, 
participants  with  LOEC  in  their  visualization  spent  more  of  their  time  modifying  paths  (M  — 
43.36%,  SD  —  18.55)  than  those  that  did  not  (M  =  24.09%,  SD  =  25.10).  This  effect  was  not  seen 
with  the  other  cost  functions  (Distance,  Metabolic,  nor  Exploration).  Therefore,  the  LOEC  groups 
trend  observed  in  Figure  6.15  is  mostly  due  to  the  difference  in  percent  time  spent  modifying  only 
within  the  Time  cost  function. 

From  Figure  6.15,  it  also  appears  that  within  each  visualization,  cost  functions  do  not  seem  to  be 
treated  equally,  mainly  for  the  groups  that  had  elevation  contours  in  their  visualization.  A  simple 
main  effects  due  to  cost  function  within  each  visualization  group  did  not  reveal  any  significant 
differences.  However,  there  was  a  significant  increase  of  percent  time  spent  modifying  the 
Exploration  cost  function  within  the  participants  that  had  the  elevation  contours  visualization. 
Within  the  elevation  contours  visualization  group,  point-wise  comparisons  showed  that  these 
participants  had  significantly  higher  percent  modifying  times  for  Metabolic  and  Exploration  cost 
functions  as  compared  to  the  Time  function.  Participants  with  the  visualization  group  with  both 
LOEC  and  elevation  contours  spent  significantly  higher  percent  modifying  times  for  Exploration 
than  Metabolic 1 .  Thus,  for  these  participants,  the  Exploration  function  was  viewed  as  challenging, 
requiring  more  modifying  time.  Additionally,  this  difference  within  visualization  supports  the 
previous  analysis  in  which  indicated  that  participants  who  only  had  the  elevation  contours 
visualization  spent  less  of  their  time  modifying  the  Time  cost  function  path. 

In  summary,  there  was  no  main  effect  between  visualization  groups  and  cost  functions  with  respect 
to  percent  time  spent  modifying  paths.  However,  there  are  some  trends  based  on  visualization. 
Only  within  the  Time  cost  function  were  there  any  significant  differences  between  visualizations: 
participants  that  had  the  LOEC  visualization  tended  to  spend  more  of  their  time  conducting 


1  Point-wise  comparisons  for  Exploration  cost  function  between  each  of  the  three  visualization 
groups,  did  not  reveal  any  differences. 
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sensitivity  analysis  on  their  paths  as  compared  to  those  that  did  not  have  this  visualization. 
Participants  with  just  the  elevation  contours  visualization  tended  to  spend  more  of  their  time 
modifying  paths  with  the  Metabolic  and  Exploration  cost  functions  than  the  Time  function. 


6.3.3.2  Phase  2 

Box-plots  of  the  percent  time  spent  modifying  in  the  second  phase  of  the  experiment  are  shown  in 
Figure  6.16.  A  repeated  measure  analysis  of  variance  (2x2x3,  scenario  x  cost  function  x 
visualization,  Figure  6.17)  was  used  to  investigate  the  differences  among  the  percent  time  spent 
modifying  paths.  There  was  no  main  effect  of  cost  function,  scenario,  or  visualization  group.  There 
were  no  significant  interactions  except  for  the  interaction  between  cost  function  and  visualization 
group  (p  =  0.026). 


Figure  6.16  Box-plot  of  percent  time  spent  modifying  paths  between  visualizations  and 

scenario  type,  Phase  2 
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Figure  6.17  Means  plot  of  percent  time  spent  modifying  across  visualization,  cost  function, 
and  scenario  type  (nominal,  left;  off-nominal,  right) 

The  significant  interaction  deserves  further  analysis.  The  estimated  means  plot  (Figure  6.17)  seems 
to  reveal  a  change  in  strategy  between  cost  functions  that  depends  on  the  visualization  group.  A 
simple  effect  analysis  of  function  within  visualization  revealed  that  there  was  a  marginally  significant 
difference  between  functions  within  elevation  contours  visualization  (p  =  0.054)  and  a  marginally 
significant  difference  within  LOEC  visualization  (p  =  0.061)  \  Therefore,  the  detected  significant 
interaction  between  cost  function  and  visualization  occurs  because  the  participants  within  the 
elevation  contours  visualization  spent  a  larger  percent  time  modifying  the  Exploration  cost  function 
than  the  Time  function  while  the  participants  within  the  LOEC  visualization  did  the  opposite, 
spending  a  larger  percent  time  modifying  the  Time  cost  function  than  the  Exploration. 

In  summary,  no  main  effect  was  found  between  scenarios,  cost  function,  nor  visualization  for 
percent  time  spent  modifying.  Of  particular  interest  is  the  lack  of  difference  across  scenarios,  as  it 
was  expected  that  degraded  automation  would  prompt  participants  to  conduct  more  sensitivity 
analysis  as  the  path  costs  presented  by  the  interface  could  not  be  trusted.  There  was  a  significant 
interaction  between  function  and  visualization,  in  which  the  elevation  contours  participants  spent 
more  time  modifying  Exploration  paths  than  Time,  while  the  LOEC  participants  tended  to  act  in 


1  There  was  no  significant  difference  to  report  for  the  third  visualization  group. 
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the  opposite  fashion.  This  was  the  same  effect  found  in  Phase  1,  though  it  was  more  pronounced 
with  the  addition  of  the  off-nominal  scenario.  The  participants  that  had  both  visualizations  spent 
equal  amount  of  their  time  modifying  both  types  of  functions,  regardless  of  scenario. 

6.3.4  True  Time 

True  time  is  a  dependent  measure  that  was  derived  from  the  path  cost  profiles  (all  path  costs 
through  the  entirety  of  a  single  trial).  True  time  is  how  long  a  participant  took  to  arrive  to  the 
minimum  path  cost  found.  It  differs  from  total  time  or  time  penalty  because  it  reflects  the  actual 
time  that  it  took  a  participant  to  optimize  a  path,  excluding  any  time  spent  afterwards  attempting  to 
find  another  solution.  Short  true  times  indicate  that  participants  were  able  to  optimize  the  path 
quickly. 

6.3.4.1  Phase  1 

Figure  6.18  shows  the  range  of  true  times  between  visualizations  and  cost  functions.  A  repeated 
measures  ANOVA  (3  x  4,  visualization  x  cost  function,  Figure  6.19)  only  revealed  a  significant 
difference  between  cost  functions  (p  <  0.0001).  There  were  no  significant  differences  across 
visualization  and  no  significant  interaction  between  treatments.  It  appears  from  Figure  6.19  that  the 
Time  cost  function  had  the  longest  tme  time.  Simple  contrasts  confirmed  that  Time  cost  function 
had  significantly  longer  tme  times  than  any  of  the  other  cost  functions. 


139 


500- 


400- 


300- 


<D 

3 


200- 


100- 


0- 


Hr 


028 


JL 


028 


08 


T  T 


Visualization  Group 
I]  Elevation  Contours 
□  LOEC 

—  Elevation  Contours  & 
U  LOEC 


— I - 1 - 

Time  Metabolic 

Function 


T 


“ 

033 


- 1 - 

Exploration 


Distance 


Figure  6.18  Box-plot  of  true  times  between  visualizations  and  cost  functions,  Phase  1 


Figure  6.19  Means  plot  of  true  time  across  visualizations  and  cost  functions,  Phase  1 
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6.3.4.2  Phase  2 


The  range  of  true  times  between  scenarios,  visualization  and  cost  functions  can  be  seen  in  Figure 
6.20.  A  repeated  measures  ANOVA  (2x2x3,  scenario  x  cost  function  x  visualizations.  Figure 
6.21)  revealed  no  main  effect  based  on  visualization,  but  there  was  a  significant  difference  between 
cost  functions  (p  <  0.0001)  and  scenario  (p  =  0.04).  There  were  no  significant  interactions.  The 
difference  between  cost  function  is  the  same  as  seen  in  the  nominal  case,  i.e.,  Time  had  longer  tme 
times  than  the  Exploration  function.  With  respect  to  scenario,  the  nominal  true  times  were  longer 
than  the  off-nominal  ones.  For  the  Time  cost  function,  the  decrease  was  on  average,  30  seconds, 
while  for  Exploration,  it  was  20  seconds. 
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Figure  6.20  Box-plot  for  true  time  between  visualization  and  scenario  type,  Phase  2 
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Figure  6.21  Means  plot  of  true  time  across  visualizations,  cost  functions  and  scenarios, 

Phase  2  (Nominal,  left;  off-nominal,  right) 


6.3.5  Differential  Cost 

Differential  cost  is  a  dependent  measure  calculated  from  the  path  cost  profiles.  It  is  the  path  cost 
error  difference  (in  percent)  between  the  first  path  cost  the  participant  made  for  a  particular  trial  and 
the  submitted  path  cost.  This  metric  helps  to  assess  how  much  path  cost  error  decreased  during  the 
optimization  process.  A  negative  differential  cost  would  indicate  that  the  participant  submitted  a 
path  that  was  worse  than  the  first  one  they  had  made.  A  small  differential  cost  would  indicate  that 
the  participant’s  first  path  attempt  was  close  in  cost  to  the  submitted  least-cosdy  path. 


6.3.5. 1  Phase  1 

The  range  of  differential  costs  for  the  different  cost  functions  across  visualization  can  be  seen  in 
Figure  6.22.  ANOVA  assumptions  were  not  met,  thus  non-parametric  tests  were  used. 
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Figure  6.22  Box-plot  of  differential  costs  between  visualizations  and  cost  functions,  Phase  1 

There  was  only  one  participant  that  had  a  large  negative  differential  cost1  within  the  Metabolic  cost 
function,  likely  a  mistake  on  selecting  a  path  to  submit.  There  were  participants  with  zero 
differential  costs,  meaning  that  these  participants  submitted  the  first  path  they  created  (2  in  Distance 
function,  4  in  Time  function,  5  each  for  Metabolic  and  Exploration  functions).  The  number  of 
participants  with  zero  differential  costs  was  counted,  and  a  Pearson’s  chi-square  test  did  not  show  a 
difference  in  distribution  between  visualization  groups  among  the  participants  who  had  zero 
differential  costs. 


Kruskal-Wallis  tests  within  each  cost  function  revealed  only  a  marginally  significant  differential  cost 
between  visualizations  for  the  Exploration  cost  function  (p  =  0.082)  and  none  for  the  other  three 
functions.  Comparisons  between  visualizations  (Mann-Whitney  tests)  showed  that  the  LOEC 


1  There  were  three  additional  subjects  with  negative  differential  costs  but  these  were  at  most  -0.5%, 
i.e.,  essentially  zero. 
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visualization  group  had  a  significantly  lower  differential  cost  than  the  visualization  group  with 
elevations  contours  &  LOEC  (p  -  0.02).  No  difference  was  detected  between  elevations  contours 
group  and  the  combined  visualization  group. 

A  Friedman  test  was  used  to  determine  a  significant  difference  in  ranking  between  the  cost 
functions,  regardless  of  visualization  (p  <  0.0001).  All  pair-wise  comparisons  (using  Wilcoxon  Sign 
tests)  were  statistically  significant  (p  <  0.0001)  except  between  Distance  and  Metabolic  cost 
functions.  This  implies  that  Time  trial  had  the  larges  differential  cost.  Exploration  had  the  smallest, 
and  there  was  no  statistical  difference  between  the  differential  costs  for  Distance  and  Metabolic 
functions. 

In  summary,  Distance  and  Metabolic  had  overall  the  same  differential  cost,  while  Time  had  the 
largest.  In  the  cost  function  with  the  smallest  differential  cost.  Exploration,  there  is  a  marginal  main 
effect  of  visualization.  Comparisons  revealed  that  participants  with  just  the  LOEC  visualization  had 
a  significantly  smaller  differential  cost  than  the  participants  in  the  elevation  contours  &  LOEC 
visualization  group.  This  implies  that  these  participants  started  out  making  better  paths  for  the 
Exploration  cost  function  than  the  combined  visualization  group. 

6.3.5.2  Phase  2 

Figure  6.23  show  box-plots  of  the  differential  cost  range  found  for  the  two  cost  functions  tested  in 
the  nominal  and  off-nominal  scenarios.  In  the  off-nominal  case  there  was  a  greater  number  of 
negative  differential  costs,  indicating  the  paths  that  were  submitted  were  higher  in  cost  than  the  first 
initial  path  cost.  In  the  Time  off-nominal  trial,  6  participants  had  negative  differential  costs,  while  in 
the  Exploration  off-nominal,  12  participants.  The  number  of  participants  with  negative  differential 
costs  (Figure  6.24)  was  counted  for  each  visualization  group.  No  significant  difference  was  detected 
in  the  number  of  participants  within  each  visualization  with  negative  differential  costs.  This  implies 
that  negative  differential  costs  among  the  off-nominal  trials  cannot  be  attributed  solely  to 
visualization  group. 

Figure  6.24  also  depicts  the  number  of  participants  that  had  a  differential  cost  of  zero,  indicating 
that  they  submitted  the  first  path  they  made  as  their  optimized  solution,  possibly  not  optimizing  at 
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all.  In  the  Time  off-nominal  trial,  there  were  6  participants  that  had  differential  costs  of  zero,  while 
in  the  Exploration  off-nominal  trial,  there  were  only  3.  There  was  no  statistically  significant 
difference  across  scenarios  for  the  number  of  participants  with  zero  differential  costs.  These 
participants  were  also  evenly  distributed  between  visualizations.  Thus,  similar  to  negative 
differential  costs,  zero  differential  costs  in  the  off-nominal  trials  cannot  be  solely  attributed  to 
visualization. 
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Figure  6.23  Box-plot  of  differential  costs  between  visualization  and  scenario  type,  Phase  2 
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Differential  Cost  Differential  Cost 

Figure  6.24  Histogram  by  visualization  group  of  number  of  participants  with  negative  or 
zero  differential  costs  in  off-nominal  trials  (left,  Time  cost  function;  right,  Exploration  cost 

function) 

Using  Kruskal-Wallis  tests,  no  differences  between  visualizations  were  detected  in  the  off-nominal 
trials  with  respect  to  differential  cost.  This  result  differs  from  the  nominal  case  where  a  marginally 
significant  difference  was  found  for  the  Exploration  function.  The  difference  between  off-nominal 
cost  functions  was  the  same  as  in  Phase  1,  and  the  Time  cost  function  had  overall  significantly  larger 
differential  costs  (Wilcoxon  Sign  test,  p  =  0.019).  Within  each  visualization,  however,  no  significant 
differences  across  cost  functions  were  detected. 

Wilcoxon  Sign  tests  were  used  to  compare  differential  costs  across  scenarios.  Participants,  as  a 
whole,  significantly  decreased  their  differential  costs  in  the  off-nominal  condition  within  Time  cost 
function  across  (p  =  0.008).  However,  this  trend  was  not  exhibited  within  the  Exploration  cost 
function  where  there  was  no  significant  difference  between  scenarios.  Within  the  elevation  contours 
visualization  group,  there  was  a  decrease  in  differential  costs  across  scenarios  (marginally  significant, 
p  =  0.062,  for  the  Time  function  and  significant,  p  =  0.01,  for  the  Exploration  function).  Within  the 
LOEC  visualization  group,  no  significant  difference  was  detected  between  scenarios  for  the  Time 
cost  function,  yet  there  was  a  marginal  significant  (p  =  0.074)  increase  across  scenarios  within 
Exploration  function.  Within  the  third  visualization  group  (both  elevation  contours  and  LOEC), 
there  was  no  difference  in  differential  costs  across  scenarios. 
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In  summary,  within  the  off-nominal  trials,  no  differences  in  differential  costs  were  found  between 
visualizations.  Across  cost  functions,  the  Time  function  had  higher  differential  costs  than 
Exploration  function,  which  is  consistent  with  Phase  1.  Across  scenarios,  there  was  a  significant 
decrease  in  mean  differential  costs  for  Time  function  but  no  difference  in  Exploration.  A  decrease 
of  differential  cost  for  the  off-nominal  case  would  indicate  some  learning  (as  participants  would 
have  been  initially  close  to  their  solution).  However,  there  was  an  increase  in  negative  differential 
costs  for  the  off-nominal  condition  (from  1%  to  26%),  meaning  that  these  poor  performers 
decreased  the  differential  costs. 

6.3.6  Non-optimal  Satisficing 

Non-optimal  satisficing  refers  to  actions  taken  by  a  participant  attempting  to  find  a  lower  path  cost 
after  a  minimum  was  already  achieved.  Specifically,  non-optimal  satisficing  is  defined  as  1)  cost 
surplus:  the  difference  between  the  minimum  path  cost  achieved  and  the  submitted  path  cost,  and  2) 
time  surplus:  the  percent  of  time  spent  between  those  ([total  time  -  true  time] /total  time).  For  non- 
optimal  cost  surplus,  non-parametric  tests  were  used.  For  non-optimal  percent  time  surplus, 
ANOVA  was  implemented.  If  the  sphericity  assumption  was  not  met,  the  Greenhouse-Geisser 
adjusted  p-value  is  reported. 

6.3.6.1  Phase  1 

The  range1  of  non-optimal  satisficing  cost  surplus  is  seen  in  Figure  6.25.  In  Phase  1,  there  was  little 
difference  between  the  tme  minimum  cost  achieved  and  submitted  cost.  The  median  for  all 
conditions  was  zero.  Hence,  participants  generally  submitted  the  minimum  path  cost  they  had 
already  achieved.  No  further  analysis  was  conducted  on  this  measure  for  this  Phase  1. 


1  Three  data  points  in  the  elevation  contours  group  were  identified  as  outliers  (4.07*SD,  3.71*SD, 
and  5.59*SD  above  the  mean)  and  hence,  removed. 
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Figure  6.25  Box-plots  of  non-optimal  satisficing,  cost  surplus,  between  visualizations  and 

cost  functions,  Phase  1 

The  range  of  non-optimal  satisficing  for  percent  of  time  surplus  can  be  seen  in  Figure  6.26.  A 
repeated  measures  3x4  ANOVA  was  used  to  test  the  differences  between  visualization  and  cost 
functions  (Figure  6.27).  A  main  effect  was  found  for  cost  function  (p  -  0.046)  but  none  for 
visualization.  There  were  no  significant  interactions.  A  simple  contrast  showed  that  Time  cost 
function  had  a  significantly  smaller  percent  time  surplus  in  non-optimal  satisficing  when  compared 
to  Metabolic  (p  =  0.001)  and  Exploration  (p  =  0.018),  but  not  Distance.  No  other  significant 
differences  were  detected  between  cost  functions. 
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Figure  6.26  Box-plot  of  non-optimal  satisficing  (percent  time  surplus)  between 
visualizations  and  cost  functions,  Phase  1 


Figure  6.27  Means  plot  for  non-optimal  satisficing  (percent  time  surplus)  between 
visualization  and  cost  functions,  Phase  1 
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It  is  interesting  to  note  that  for  the  Exploration  cost  function,  the  mean  percent  time  surplus  in  non- 
optimal  satisficing  increases  for  just  the  elevation  contours  group.  A  pair-wise  comparison  within 
this  visualization  indicated  significant  differences  between  Exploration  and  both  Time  (p  =  0.055) 
and  Distance  (p  =  0.041). 

In  summary,  for  Phase  1 ,  participants  did  not  submit  sub-optimal  path  costs  relative  to  the  achieved 
minimum.  They  did  however,  spend  on  average  between  20  —  35%  of  their  time  conducting  non- 
optimal  satisficing.  The  lowest  of  these  percent  time  surplus  was  found  within  the  Time  cost 
function,  which  was  significantly  lower  than  Metabolic  and  Exploration  trials.  This  difference 
between  cost  functions  for  percent  time  surplus  in  conjunction  with  the  previous  result  where  Time 
function  had  the  largest  true  time,  suggests  that  participants  were  feeling  the  time  pressure  of 
completing  their  optimized  path  as  fast  as  possible.  In  other  words,  after  spending  a  long  time 
attempting  to  solve  the  Time  function  path,  participants  did  not  spend  additional  time  conducting 
non-optimal  satisficing  once  a  minimum  path  was  found.  This  is  supported  by  a  significant 
correlation  within  the  Time  function  between  non-optimal  satisficing  time  surplus  and  true  time 
(Pearson  correlation  =  -0.48,  p  =  0.005). 

6.3.6.2  Phase  2 

The  range1  of  non-optimal  satisficing  cost  surplus  measures  (cost  differences  between  minimum 
cost  found  and  cost  submitted)  are  shown  in  Figure  6.28.  The  non-parametric  test,  Kruskal-Wallis, 
did  not  detect  a  significant  difference  between  visualizations  within  the  off-nominal  scenario. 
Comparisons  across  scenarios  (Wilcoxon  Sign  tests)  revealed  that  there  was  a  significant  increase  in 
cost  surplus  for  both  Time  (p  =  0.032)  and  Exploration  (p  =  0.011),  meaning  the  cost  surplus  was 
higher  for  both  functions  in  the  off-nominal  scenario.  This  increase  in  cost  surplus  was  about 
seven-fold  for  each  of  the  cost  functions.  Within  the  off-nominal  scenarios,  paths  made  in  the  Time 
cost  function  had  a  significantly  higher  cost  surplus  than  Exploration  function  (Wilcoxon  Sign  test, 
p  =  0.001). 


1  Three  outliers  were  removed  (3.7 DSD,  4.15*SD,  and  4.48*SD  above  the  mean). 
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Figure  6.28  Box-plot  of  non-optimal  satisficing  (cost  surplus)  between  visualizations  and 

scenarios,  Phase  2 

The  range  of  the  percent  time  spent  conducting  non-optimal  satisficing  is  shown  in  Figure  6.29. 
Differences  in  scenario,  cost  function,  and  visualization  were  tested  using  a  repeated  measure 
ANOVA  (2x2x3,  scenario  x  cost  function  x  visualization,  Figure  6.30).  There  was  a  significant 
difference  between  cost  functions  (p  =  0.007),  where  the  percent  time  surplus  is  smaller  for  the 
Time  cost  function  than  Exploration,  which  is  the  same  effect  seen  in  Phase  1.  No  difference  across 
visualization  was  detected  and  no  interactions  were  significant.  There  was  a  significant  difference 
across  scenario  (p  =  0.033),  meaning  that  in  the  off-nominal  scenarios,  the  percent  time  spent  in 
non-optimal  satisficing  increased.  There  was  a  significant  correlation  between  time  surplus  and  tme 
time  (p  <  0.0001). 
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Figure  6.29  Box-plot  for  non-optimal  satisficing  (percent  time  surplus)  between 
visualizations  and  scenarios,  Phase  2 
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Figure  6.30  Means  plot  of  non-optimal  satisficing  (percent  time)  between  visualizations, 
cost  functions,  and  scenario  (nominal,  left;  off-nominal,  right) 

In  summary,  across  scenarios,  more  non-optimal  satisficing  occurred  from  nominal  to  off-nominal 
scenarios.  Under  degraded  automation  conditions,  participants  introduced  cost  errors  not  seen  in 
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die  nominal  case  which  is  represented  in  the  seven-fold  increase  of  cost  surplus  in  the  off-nominal 
scenarios.  Within  the  off-nominal  trials,  the  Time  cost  funcdon  had  a  significantly  larger  cost 
surplus  as  compared  to  the  Exploration  function.  With  respect  to  percent  time  surplus.  Time  cost 
function  trials  had  a  smaller  percent  time  spent  non-optimal  satisficing  (compared  to  Exploration) 
and  no  difference  in  visualization  was  detected.  Across  scenarios,  there  was  a  significant  increase  in 
non-optimal  percent  time  surplus  for  the  off-nominal  trials.  In  the  off-nominal  case,  participants 
generally  spent  the  same  percent  time  modifying  paths  across  scenarios,  but  did  not  realize  that  they 
had  reached  their  minimum  path  cost,  resulting  in  an  increase  of  non-optimal  time  surplus. 

6.3.7  Other  Analyses 

There  are  a  few  other  analyses  worth  mentioning  that  were  conducted  to  further  investigate  how 
participants  performed  in  this  second  experiment. 

6.3.7. 1  Path  Cost  Errors  and  Total  Time  Correlations 

Correlations  between  path  cost  errors  and  time  (i.e.,  time  penalty,  true  time,  and  actual  time)  were 
conducted  in  order  to  assess  if  longer  time  on  the  optimization  task  was  proportional  to  a  decrease 
in  errors.  The  path  cost  errors  were  not  normally  distributed  (Kolmogorov-Smirnov  test,  p  < 
0.0001).  Furthermore,  time  penalty  and  true  time  were  not  normally  distributed  (p  -  0.001  and  p  = 
0.037,  respectively).  Thus,  correlations  were  analyzed  using  Spearman’s  rho  correlations  (equivalent 
non-parametric  test  for  Pearson’s  correlation).  There  were  no  significant  correlations  between 
errors  and  any  of  the  time  measures. 

63.1.2  Preference  Between  Visualizations 

Within  the  group  that  had  both  visualizations,  elevation  contours  and  LOEC,  participants  were 
asked  if  they  preferred  the  elevation  contours  more  than  the  LOEC.  Eleven  participants  answered 
this  question  (Figure  6.31),  and  no  statistical  difference  between  preferences  (%2(3,  11)  =  0.15,  p  = 
0.96). 
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Figure  6.31  Histogram  of  response  for  visualization  preference  within  group  that  had  both 

elevation  contours  and  LOEC. 

6.3.7.3  Cost  Function  Tab 

Every  interaction  between  the  participant  and  the  PATH  interface  (every  click)  was  recorded,  thus 
total  number  of  clicks  per  interface  element  (i.e.,  a  particular  button)  could  be  assessed.  This  could 
potentially  provide  insight  to  what  elements  of  the  interface  were  used  often  or  not  at  all.  Of 
particular  interest  was  the  number  of  times  the  cost  function  tab  (top  left  corner  of  interface,  Figure 
6.1)  was  clicked  because  a  high  count  would  indicate  that  the  participant  needed  detailed 
information  about  the  cost  function  itself.  In  general,  this  information  was  not  accessed  during  the 
test  trials  though  it  was  for  the  practice  trials,  regardless  of  visualization.  Even  in  the  practice  trials, 
the  number  of  times  is  small.  For  the  off-nominal  trials,  most  participants  did  not  click  on  this  tab 
either. 


6.4  Cognitive  Strategies 

Based  on  observation,  participant  debriefs,  and  experimental  data,  the  following  path  planning 
strategies  were  surmised.  As  in  the  first  experiment,  most  participants  typically  created  one  or  two 
paths  and  then  proceeded  to  modify  waypoints,  i.e.,  manual  sensitivity  analysis.  However,  it  was 
observed  that  there  were  some  participants  that  chose  not  to  modify  their  paths  much,  if  at  all.  This 
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is  not  say  these  participants  did  not  do  sensitivity  analysis,  but  rather  conducted  a  “whole-path” 
sensitivity  analysis  by  making  multiple  path  alternatives. 

If  the  percent  time  spent  modifying  paths  (a  measure  of  the  participant’s  manual  sensitivity  analysis) 
is  averaged  per  participant  over  all  six  test  trials,  a  bimodal  distribution  emerges  (Figure  6.32). 
Participants  that  conducted  manual  sensitivity  analysis  (modified  path  by  adding,  deleting,  and/or 
moving  waypoints)  are  the  ones  with  average  modifying  times  greater  than  16%,  which  is  three- 
fourths  of  the  total  number  of  participants.  Of  the  participants  that  conducted  “whole  path” 
sensitivity  analysis  (with  average  modifying  times  less  than  16%),  five  were  in  the  elevation  contours 
group,  two  were  in  the  LOEC  visualization  group,  and  only  one  was  in  the  third  visualization  group. 
Using  a  Pearson’s  chi-square  test,  no  statistically  significant  difference  in  participant  distribution 
across  visualizations  was  detected  (%2(2,34)  =  3.64,  p  =  0.16).  Additionally,  Mann-Whitney  tests  did 
not  detect  any  performance  differences  (with  respect  to  path  cost  errors)  between  the  participants 
that  chose  manual  versus  “whole  path”  sensitivity  analysis. 


Average  Percent  Time  Modifying 


Figure  6.32  Histogram  of  average  percent  time  modifying  across  all  trials 

Of  the  participants  that  did  conduct  manual  sensitivity  analysis,  they  primarily  focused  on  modifying 
single  waypoints.  Multi-waypoint  modifications  were  observed,  but  it  was  not  the  most  prominent 
strategy.  During  the  experimental  debriefs,  some  participants  mentioned  that  waypoints  were 
moved  or  added  based  on  the  visualization  presented.  Those  participants  that  had  LOEC,  for 
example,  would  try  to  move  waypoints  into  areas  that  were  “more  yellow”  (smaller  total  cost  to  the 
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goal)  and  move  away  from  “purple”  (higher  total  cost  to  the  goal).  For  those  participants  that  had 
elevation  contours,  they  would  try  to  keep  waypoints  between  contours,  in  essence,  attempting  to 
make  “flat”  paths  (with  small  elevation  changes).  During  degraded  automation,  participants,  mainly 
those  with  LOEC  visualization,  mentioned  in  their  debriefs  that  they  leveraged  the  path  elevation 
profile  to  make  their  least-costly  paths.  In  this  case,  the  waypoints  were  added  and  moved  based  on 
path  elevation  profiles,  where  participants  were  attempting  to  make  their  paths  “flatter”,  i.e., 
reducing  changes  in  elevations. 

Aside  from  using  elevation  contours  and  the  LOEC  colors  to  make  and  modify  paths,  many 
participants  commented  on  also  having  used  the  simple  heuristic  of  “shortest  path”.  For  example, 
an  LOEC  participant  summarized  his/her  strategy  as  attempting  to  make  the  shortest  path  to  a 
yellow  area.  The  choice  of  this  heuristic  is  not  surprising  because,  even  though  all  the  cost  functions 
(aside  from  Distance)  had  some  relationship  to  slope:  in  general  the  longer  the  path,  the  higher  its 
cost.  Furthermore,  this  is  representative  of  a  “fast  and  frugal”  heuristic  (Chase,  Hertwig,  & 
Gigerenzer,  1998;  Gigerenzer  &  Goldstein,  1996),  where  the  operator  implements  a  heuristic  that 
he/ she  recognizes  quickly  or  considers  simple.  For  the  case  of  path  planning,  finding  the  shortest 
distance  is  a  task  that  is  perceptually  driven  and  which  humans  perform  well  (see  also  section  2.2), 
and  hence,  is  a  relatively  simple  and  familiar  strategy  to  implement. 

Testing  if  participants  utilized  the  simple  heuristic  or  strategy  of  shortest  distance,  particularly  for 
the  more  complex  cost  functions  such  as  Metabolic  and  Exploration,  can  be  quantified.  This  can  be 
done  by  comparing  a  participant’s  solution  to  an  objective  shortest  path  solution.  If  the  difference 
between  these  costs  is  small,  then  the  participant  was  classified  as  having  implemented  a  shortest 
path  heuristic.  Thus  for  all  trials,  the  shortest  path  distance  was  determined  and  every  path 
submitted  as  a  solution  by  the  participants  per  test  trial  was  retrospectively  recalculated  for  the 
Distance  cost  function  (i.e.,  the  length  of  each  test  trial  path).  Comparing  the  length  of  submitted 
path  to  the  shortest  distance  possible  resulted  in  a  distance  error  (in  the  case  for  Distance,  this  is  the 
same  path  cost  error  found  for  this  trial). 

Comparisons  between  conditions  were  done  using  a  repeated  measures  ANOVA  (see  Appendix  B 
for  details).  Within  the  nominal  trials  (left.  Figure  6.33),  there  was  a  significant  difference  across 
cost  functions  (p  <  0.0001).  Simple  contrasts  reveal  that  the  distance  errors  for  all  the  cost 
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functions  were  significantly  larger  than  Distance  (Time,  p  <  0.0001;  Metabolic,  p  <  0.0001; 
Exploration,  p  =  0.011).  All  other  comparisons  were  significant  (p  <  0.0001)  except  between  Time 
and  Metabolic  (p  =  0.40).  These  results  indicate  that  participants  were  implementing  other 
heuristics  aside  from  the  shortest  path  strategy  for  the  other  cost  functions,  though  at  a  lesser  extent 
for  the  Exploration  cost  function. 


Figure  6.33  Means  plot  for  distance  errors  (right,  nominal  cost  functions;  left,  off-nominal 

cost  functions  across  scenarios) 

Distance  errors  were  compared  across  scenarios  in  order  to  determine  if  participants  consistently 
applied  a  shortest  path  heuristic.  For  the  off-nominal  conditions,  there  was  a  significant  difference 
across  cost  function  (p  <  0.0001)  and  a  marginal  difference  across  scenario  (p  =  0.09),  though  there 
was  a  significant  interaction  between  these  conditions  (p  =  0.004).  Point- wise  comparisons  showed 
that  there  was  a  significant  difference  across  scenarios  within  the  Time  cost  function  only  (p  = 
0.003)  and  not  Exploration  (p  =  0.36).  This  results  means  that  under  off-nominal  conditions,  there 
was  no  change  in  heuristic  for  the  Exploration  trial,  while  for  Time,  participants  tended  to  make 
shorter  paths  and  thus  relying  on  the  shortest  heuristic  more1. 


1  A  similar  test  was  conducted  for  a  second  possible  heuristic:  “following  contours”.  For  each 
subject  and  trial,  the  submitted  path  was  categorized  as  following  or  not  a  contour  (elevation  or 
LOEC,  depending  on  the  visualization).  Some  similar  results  as  these  presented  emerged,  which  was 
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6.5 


Pareto  Front  Analysis 


The  two  key  measures  of  path  planning  performance  in  this  experiment  were  path  cost  errors  and 
total  task  time.  In  order  to  better  understand  the  relationship  between  these  measures,  individual 
participants  along  a  Pareto  front  were  assessed.  The  Pareto  front  is  the  set  of  non-dominated 
solution  points;  in  this  case,  if  comparing  task  time  and  cost  error,  a  non-dominated  point  would  be 
the  smallest  error  achieved  at  a  particular  task  time.  Additionally,  examining  the  Pareto  front  will 
help  understand  the  effect  of  visualization,  cost  function,  and  scenario  in  terms  of  the  best 
performers  as  opposed  to  identifying  the  main  effects.  Since  the  task  was  time  pressured,  assessing 
how  participants  conducted  the  path  cost-time  trade  would  be  beneficial  to  the  overall 
understanding  of  the  human  optimization  process. 

Participant  performance  was  plotted  with  respect  to  total  task  time  and  path  cost  errors.  A 
participant  is  included  in  the  Pareto  front  if  no  other  participant  was  able  to  reach  a  smaller  path 
cost  error  in  a  faster  time.  It  is  worth  noting  that  occurrence  in  the  Pareto  front  does  not  necessarily 
mean  that  tire  participant  performed  overall  well.  The  best  participant  would  be  one  that  had  a  very 
small  cost  error  in  a  short  period  of  time. 

6.5.1  Nominal  Scenario 

The  scatter  plot  of  path  cost  errors  versus  total  time  was  generated  for  each  nominal  cost  function 
in  order  to  determine  which  participants  were  on  the  Pareto  front.  The  Pareto  fronts  for  the 
Distance  and  Metabolic  cost  functions  (Figure  6.34)  were  “flat”,  meaning  that  the  participants  along 
the  front  have  relatively  the  same  path  cost  with  increasing  total  time.  In  other  words,  regardless  of 
the  time  spent  on  the  optimizing  task,  path  cost  errors  do  not  improve  much.  This  is  expected  from 
the  Distance  cost  function,  which  was  the  easiest  cost  function  for  participants  to  optimize.  In  the 
Distance  Pareto  front,  there  were  participants  from  all  three  visualizations.  In  the  Metabolic  Pareto 
front,  there  were  two  elevation  contours  participants  with  path  cost  errors  of  5.2%  and  4.2% 


not  unexpected  participants  often  applied  both  types  of  heuristics  (shortest  path  and  following 
contours). 
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(subjects  5  and  19  in  Figure  6.34),  taking  59  and  93  seconds,  respectively.  These  two  participants  did 
not  spend  any  time  modifying  paths  (just  making  a  few  whole  paths).  The  third  participant  (subject 
8)  in  the  Metabolic  Pareto  front  had  the  smallest  error  4.0%  (in  the  LOEC  group),  however  he/ she 
spent  256  seconds  on  the  task  and  75%  of  it  modifying  the  path.  For  the  Metabolic  cost  function, 
the  best  performers  were  in  the  elevation  contours  and  additional  manual  sensitivity  analysis  helped 
decrease  the  path  cost  errors  but  not  by  a  large  amount. 
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Figure  6.34  Scatter  plot  of  path  cost  errors  and  total  time  for  Distance  (left)  and  Metabolic 

(right)  cost  functions 

The  Time  and  Exploration  cost  functions  have  a  clearer  Pareto  front  (Figure  6.35).  The  set  of 
Pareto  points  for  the  Time  cost  function  (left.  Figure  6.35)  identified  three  participants  in  this  non- 
dominated  front,  all  in  the  elevation  contours  visualization  group.  In  decreasing  order,  the  cost 
errors  drop  from  37.1%  to  33.1%  to  26.2%  (subjects  17,  10,  and  20,  respectively1).  Compared  to 
the  first  participant,  total  times  spent  completing  the  task  triples  and  quintuples  for  improvements  of 
4%  and  11%.  The  trend  with  respect  to  percent  time  spent  modifying  was  0%  to  56%  to  62%. 
Thus,  the  main  difference  between  these  participants  is  the  amount  of  time  spent  making  single 


1  It  is  important,  though,  to  note  that  the  average  path  cost  error  for  this  trial  was  60%. 
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waypoint  modifications.  Increased  time  and  modifications  resulted  in  decreased  cost  error.  It  is 
surprising  that  subject  17  did  well  in  the  Time  cost  function,  the  trial  with  the  largest  errors;  he  or 
she  was  the  only  participant  that  self-designated  as  having  no  map  or  hiking  experience  and  did  not 
do  particularly  well  in  other  trials,  especially  the  Metabolic  trial.  Nonetheless,  the  exhibited  trend 
was  that  increased  manual  sensitivity  within  the  elevation  contours  participants  led  to  improved  path 
planning  performance. 
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Figure  6.35  Scatter  plot  of  path  cost  errors  and  total  time  for  Time  (left)  and  Exploration 

(right)  cost  functions,  nominal  cases 

In  the  Pareto  front  for  the  nominal  Exploration  cost  function  (right.  Figure  6.35),  there  was  a 
participant  that  stands  out  as  having  superior  performance  (subject  7  in  elevation  contours  group). 
However,  this  participant  performed  poorly  for  the  Time  and  Exploration  cost  functions.  Thus,  the 
Pareto  front  for  the  Exploration  function  is  considered  with  and  without  this  participant. 

If  considering  subject  7,  the  Exploration  function  Pareto  front,  three  of  the  four  participants  were  in 
the  elevation  contours  group.  A  reduction  of  path  cost  errors  was  accompanied  by  about  20-second 
increase  in  total  time  spent  optimizing.  The  path  cost  errors  along  the  Pareto  front  were  6.0%, 
5.2%,  4.8%,  and  3.6%  with  corresponding  total  task  time  of  65,  82,  100,  and  117  seconds  (subjects 
17,  31,  34,  and  7  in  Figure  6.35).  The  first  two  participants  did  no  path  modifications  while  the 
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other  two  spent  5%  and  12%  of  their  optimizing  time  modifying  their  paths.  Thus,  though  this 
trend  was  not  as  pronounced  as  in  the  Time  cost  function,  the  overall  best  Exploration  cost 
function  performers  were  found  in  the  elevation  contours  group  and  a  slight  increase  in  path 
modifications  led  to  a  decrease  in  path  cost  errors. 

If  considering  the  Exploration  Pareto  front  without  subject  7,  there  was  a  definite  change  across  the 
Pareto  front.  Of  this  set,  the  participant  with  the  smallest  path  cost  was  an  elevation  contours 
participant,  with  a  path  cost  error  of  3.9%  (subject  20).  However,  he/she  spent  263  seconds 
optimizing  the  path,  68%  of  it  conducting  path  modifications.  This  was  the  same  overall  trend 
observed  with  the  other  participants,  where  a  large  portion  of  time  was  spent  in  manual  sensitivity 
analysis,  though  without  much  decrease  in  path  cost  errors.  Additionally,  these  other  participants 
were  from  the  LOEC  visualization  groups.  This  trend  indicates  that  while  participants  increased 
their  time  optimizing  paths  through  modifications,  it  led  to  small  benefits  with  regards  to  cost.  The 
participants’  inability  to  decrease  errors  over  time  indicates  that  participants  understood  poorly  how 
to  improve  path  cost  errors  for  the  Exploration  cost  function. 

For  the  nominal  scenario,  the  general  trend  observed  for  the  Pareto  front  was  that  participants  with 
low  cost  errors  and  total  times  were  in  the  elevation  contours  visualization  group.  Furthermore,  for 
the  Time  and  (to  a  lesser  extent)  Exploration  cost  functions,  a  decrease  in  path  cost  errors  was 
accompanied  by  an  increase  in  percent  time  spent  modifying,  which  can  be  interpreted  as  an  increase 
in  manual  sensitivity  analysis. 

6.5.2  Off-Nominal  Scenario 

The  scatter  plots  of  path  cost  errors  versus  total  time  for  the  off-nominal  Time  and  Exploration  cost 
functions  are  shown  in  Figure  6.36.  The  Pareto  front  under  the  off-nominal  scenario  for  Time  was 
relatively  well-defined,  yet  for  the  Exploration,  it  appears  to  be  rather  “flat”  (i.e.,  little  decrease  in 
path  cost  errors  with  time). 

For  the  Time  off-nominal  Pareto  front  (left.  Figure  6.36),  there  was  a  decrease  in  path  cost  errors 
from  39%  to  31%  to  22%,  with  corresponding  total  times  of  57  to  88  to  163  seconds  (subjects  6,  28, 
and  20,  respectively).  Only  the  participant  with  the  lowest  error  conducted  manual  sensitivity 


161 


analysis,  with  33%  of  his/her  time  spent  modifying  paths.  The  second  participant  made  multiple 
paths,  while  the  remaining  Pareto  front  participant  made  only  one  path.  It  is  important  to  note  that, 
unlike  the  nominal  case,  the  participants  along  the  off-nominal  Time  Pareto  front  belong  to  other 
visualization  groups  (not  just  elevation  contours).  While  the  trend  was  not  as  strong  as  in  the 
nominal,  participants  decreased  path  costs  with  an  increase  in  sensitivity  analysis. 
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Figure  6.36  Scatter  plot  of  path  cost  errors  and  total  time  for  Time  (left)  and  Exploration 

(right)  cost  functions,  off-nominal  cases 

For  the  case  of  off-nominal  Exploration  (right,  Figure  6.36),  two  of  the  three  participants  in  the 
Pareto  front  were  from  the  LOEC  visualization  group.  Participants  decreased  in  errors  from  8%  to 
7.2%  to  5.6%  (subjects  17,  14  and  8,  respectively).  Correspondingly,  the  amount  of  time  spent  was 
28,  41,  and  238  seconds.  Only  the  participant  with  the  smallest  path  cost  spent  any  time  modifying 
paths  (70%). 
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Figure  6.37  Aggregate  scatter  plot  of  path  cost  errors  and  total  time  across  cost  functions, 
visualization,  and  scenario  (Pareto  front  circled) 

The  most  important  trend  observed  in  the  off-nominal  cases  was  the  presence  of  participants  that 
belonged  to  the  other  LOEC  visualizations.  In  addition,  only  the  participants  with  the  smallest  path 
cost  errors  spent  large  amounts  of  time  modifying  paths,  i.e.,  conducting  manual  sensitivity  analyses. 
The  Pareto  front  analysis  suggests  that  in  the  nominal  case,  the  best  performers  (low  path  cost  errors 
and  total  time)  are  within  elevation  contours,  while  the  LOEC  was  most  useful  in  the  off-nominal 
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cases.  If  considering  the  Pareto  front  with  all  the  independent  variables1  (circle  in  Figure  6.37), 
there  is  a  larger  representation  of  participants  in  the  elevation  contours  group  and  LOEC 
visualization  group  (and  only  two  of  the  combination  visualization  group). 

6.6  Discussion 

The  two  phases  of  this  experiment  address  two  conditions:  one  pertaining  to  path  optimization  of 
four  cost  functions  under  nominal  conditions,  and  the  other,  a  subset  of  two  cost  functions  under 
off-nominal  conditions.  In  both,  there  were  three  visualization  groups:  elevation  contours 
visualization,  levels  of  equal  cost  (LOEC)  visualization,  and  both.  As  in  the  first  experiment,  it  was 
not  practical  to  test  participants  in  all  three  of  the  visualizations.  As  a  result,  the  visualization 
method  was  a  between- subject  variable.  Participants  were  randomly  assigned  to  one  of  the 
visualization  groups,  balancing  the  number  and  gender  of  subjects  in  each.  Aside  from  map-use 
experience,  participants  were  also  asked  to  complete  an  independent  spatial  ability  test.  Neither  of 
these  metrics  resulted  in  a  viable  covariate  (i.e.,  these  measures  did  not  predict  performance). 
Furthermore,  there  was  no  statistical  evidence  that  there  was  a  visualization  group  difference  with 
respect  to  map-use  experience,  spatial  ability  test  score,  nor  choice  of  strategy  type.  Thus,  the 
participants  in  this  experiment  were  reasonably  balanced  across  the  visualization  groups. 

6.6.1  Nominal  Scenario 

Table  6.3  summarizes  the  major  results  for  the  dependent  and  independent  variables  within  Phase  1. 


1  Only  the  most  complex  functions  (Metabolic  and  Exploration)  are  shown  as  these  are  the  most 
representative  of  what  would  be  included  in  an  actual  path  planner.  The  Time  cost  function  errors 
were  significantly  larger,  creating  a  frontier  at  the  top  of  the  figure. 
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Table  6.3  Summary  of  results  for  Phase  1 


Independent  Variable 

Dependent  Variable 

Visualization 
(EC,  LOEC,  Both) 

Cost  Function 

Path  cost  errors 

No  main  effect 

Time  >  Metabolic  ~  Exploration  >  Distance 

Time  penalty 

No  main  effect 

Time  >  Distance,  Metabolic,  Exploration 

Percent  time  modifying 

No  main  effect 

Within  LOEC  groups,  Time  >  EC  Time 

True  time 

No  main  effect 

Time  >  Distance,  Metabolic,  Exploration 

Differential  cost 

Within  Exploration:  LOEC  <  Both 

Time  >  Distance  ~  Metabolic  >  Exploration 

Non-optimal  satisficing 
(cost  surplus) 

No  main  effect 

No  main  effect 

Non-optimal  satisficing 
(time  surplus) 

No  main  effect 

Time  <  Metabolic,  Exploration  ~  Distance 

6.6.1. 1  Cost  Functions 

It  was  hypothesized  that  path  planning  performance  would  decrease  with  the  number  of  variables 
being  manipulated  through  the  optimizing  cost  function,  though  this  decline  would  be  lessened  by 
the  LOEC  visualization.  Thus,  Distance  would  be  the  easiest  to  optimize,  followed  by  Time, 
Metabolic,  and  Exploration  cost  functions.  Deteriorated  performance  would  be  observed  in  an 
increase  of  errors  and  time  spent  on  path  planning  task.  Unexpectedly,  in  this  experiment,  Time  was 
the  most  difficult  cost  function  to  optimize,  as  supported  consistently  through  every  measure  of 
performance.  Difficulty  is  defined  as  having  the  highest  path  cost  errors  and  time  penalty  (total 
time).  Participants,  regardless  of  visualization  group,  had  significantly  higher  path  cost  errors,  time 
penalty  (total  time),  and  true  time  for  the  Time  cost  function.  Particularly  for  the  path  cost  errors, 
the  average  magnitude  of  the  Time  errors  was  about  ten  times  larger  than  the  hypothesized  more 
difficult  cost  functions  (Metabolic  and  Exploration). 

As  expected,  optimizing  to  the  shortest  path  (Distance)  was  the  easiest  for  participants,  regardless  of 
visualization.  Their  path  cost  errors  were  about  3  ±  3%.  This  error  doubled  for  Metabolic  and 
Exploration,  7  ±  2%  and  6  ±  2%,  respectively.  Path  cost  errors  for  the  Time  cost  functions  were  60 


165 


±  14%.  There  was  no  statistically  significant  difference  between  Metabolic  and  Exploration  path 
cost  errors.  While  there  were  differences  among  the  path  cost  errors,  no  significant  difference  was 
found  between  these  three  cost  functions  with  regards  to  time  penalty  and  true  time. 

Cost  function  differences  were  also  detected  among  the  path  cost  profiles  measures,  specifically 
differential  cost  and  percent  time  spent  conducting  non-optimal  satisficing1.  Differential  cost, 
derived  from  the  path  cost  profiles,  was  the  percent  difference  between  the  last  and  the  first  path 
cost  error;  it  is  a  measure  of  “how  close”  participants  were  in  their  first  attempt  at  optimizing  the 
path.  Again,  Time  had  the  largest  differential  cost  (26.6  ±  19.4%)  while  there  was  no  difference 
between  Distance  (7.66  ±  6.82%)  and  Metabolic  (8.82  ±  7.63%).  Surprisingly,  the  most  complex 
cost  function  presented  to  participants,  Exploration,  had  the  smallest  differential  cost  (2.97  ± 
3.05%).  These  results  imply  that  for  the  Time  cost  function,  participants’  initial  paths  were  far  from 
their  optimal  (as  compared  to  the  other  three  cost  functions).  On  the  other  hand,  participants 
started  out  relatively  close  to  their  submitted  path  solution  for  the  Exploration  cost  function.  This 
may  have  been  due  to  learning,  however,  the  fact  that  there  were  significant  differential  cost 
differences  between  Exploration  and  Metabolic  functions  even  though  there  was  no  significant 
difference  in  path  cost  error  nor  time  penalty,  implies  that  participants  were  not  able  to  improve  on 
the  path  cost  errors  for  the  Exploration  function  as  much  as  with  the  Metabolic  paths.  Thus,  while 
Time  was  a  difficult  cost  function  for  participants  to  optimize,  participants  still  found  Exploration  a 
challenging  function. 

Since  participants  spent  more  time  attempting  to  optimize  the  Time  function  paths  than  any  other 
cost  function,  it  would  be  expected  that  the  invested  time  would  result  in  small  path  cost  errors. 
Yet,  this  was  not  the  case;  participants  did  reduce  their  initial  path  errors  the  most  (i.e.,  largest 
differential  cost)  for  Time  function  but  the  cost  errors  were  the  highest.  This  would  suggest  that 
participants  could  not  improve  on  their  solution  any  more.  However,  this  is  not  necessarily  true  as 
participants  might  have  not  spent  more  time  optimizing  because  of  the  time  pressure.  This 
hypothesis  is  supported  by  the  fact  that  within  the  Time  cost  function,  participants  spent  a 
significantly  smaller  percent  of  time  in  non-optimal  satisficing  (than  in  the  Metabolic  and 
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For  cost  functions,  there  was  no  main  effect  detected  for  percent  time  spent  modifying  path. 
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Exploration  cost  functions).  Therefore,  it  is  might  be  possible  for  participants  to  reduce  path  cost 
errors  for  paths  based  on  the  Time  cost  function  if  they  had  not  been  time  pressured. 

To  sum  up  the  cost  function  effect,  based  on  participants’  path  planning  performance,  it  appears 
that  participants  viewed  the  Metabolic  and  Exploration  as  equally  difficult  cost  functions,  yet  easier 
than  the  Time  cost  function.  This  is  surprising  because  Metabolic  and  Exploration  functions  were 
more  complex  functions,  involving  three  and  four  variables,  respectively,  while  the  Time  function 
only  included  two  (Distance  and  slope).  There  is  some  evidence,  with  the  differential  cost  metric, 
that  indicates  participants  found  the  Exploration  function  challenging  (based  on  the  small 
differential  costs).  Nonetheless,  overall,  the  Time  function  was  the  most  difficult  cost  function  to 
optimize  for  all  participants. 

Performance  metrics,  however,  do  not  explain  entirely  why  the  Time  cost  function  was  the  most 
difficult  to  optimize.  One  reason  may  be  that  humans  more  intuitively  understand  how  to  optimize 
time.  For  instance,  daily  people  attempt  to  find  the  fastest  way  from  one  location  to  another,  such 
as  in  driving  tasks.  Conversely,  optimizing  a  more  complex  function  like  Metabolic  or  Exploration 
is  more  of  an  abstract  notion.  Participants  may  have  felt  they  could  optimize  a  path  based  on  Time 
as  opposed  to  Metabolic  and  Exploration,  and  hence,  spent  more  time  optimizing  the  conceptually 
easier  function  to  understand.  The  Metabolic  and  Exploration  cost  functions  may  be  so  intricate 
that  the  path  planning  task  may  have  immediately  exceeded  the  human  capacity  for  understanding 
how  to  fundamentally  optimize  these  functions.  As  a  result,  the  problem  became  “opaque”  and 
participants  relied  on  the  information  and  tools  they  were  provided  (i.e.,  path  costs  and 
visualizations)  without  giving  the  problem  as  much  thought  and  attention  as  the  Time  cost  function 
(as  exhibited  by  a  lower  time  spent  on  task).  This  would  imply  that  participants  would  revert  to  very 
simple  heuristics  for  the  more  complex  cost  functions  of  Metabolic  and  Exploration. 

Even  though  participants  repeatedly  mentioned  during  their  experimental  debrief  that  they  used  a 
“shortest  path”  heuristic,  the  results  imply  that  some  combination  heuristic  was  implemented 
because  the  submitted  paths  were  not  the  shortest  paths  possible.  Comparing  across  cost  functions, 
the  shortest  path  heuristic  seems  to  have  been  applied  more  to  the  Exploration  cost  function  than  to 
the  Metabolic  and  Time  functions.  This  is  consistent  with  the  fact  that  participants  did  not  fully 
understand  how  to  optimize  the  Exploration  cost  function.  The  Exploration  function  relies  on  sun 
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position,  and  participants  may  not  have  developed  a  good  heuristic  for  this  additional  variable.  With 
respect  to  the  Time  cost  function,  the  shortest  path  heuristic  was  an  ineffective  strategy  because  it 
does  not  take  into  account  changes  in  slope  which  are  essential  in  this  cost  function.  Therefore, 
even  though  participants  did  not  primarily  use  this  heuristic  for  the  Time  cost  function,  applying  it 
for  a  portion  of  the  path  results  in  poor  path  planning  performance. 

Poor  performance  on  the  Time  function  may  have  also  resulted  from  the  variability  and  sensitivity 
of  the  cost  function  to  changes  in  its  variables  (i.e.,  slope).  This  means  that  when  participants 
attempted  to  optimize  a  Time  path,  small  path  modifications  resulted  in  large  path  cost  changes.  To 
illustrate  this.  Figure  6.38  has  two  screen  captures,  one  for  the  Time  and  the  other  for  the 
Exploration  function,  that  contain  two  paths  each.  Below  the  map,  the  path  cost  profiles  are  also 
shown.  If  path  costs  are  compared  (in  this  case  to  the  blue  path  labeled  “base”),  one  can  see  that  a 
relatively  small  path  modification  (red  path)  results  in  a  large  cost  difference.  On  the  other  hand, 
this  is  not  seen  for  the  Exploration  function  paths,  where  a  path  modification  resulted  in  a  small 
cost  difference.  Thus,  within  the  Time  cost  function,  small  changes  in  slope  resulted  in  a  large  effect 
on  the  total  path  cost  as  compared  to  the  other  cost  function.  Cost  functions  that  exhibit  this  trait 
could  be  labeled  “overly-sensitive”  cost  function.  Due  to  the  variability,  participants  could  more 
readily  assess  the  impact  of  a  path  modification  on  the  overall  path  cost  (i.e.,  changes  were  more 
salient).  For  example,  this  can  be  observed  in  the  path  cost  profiles  for  the  Time  paths  in  Figure 
6.38.  However,  this  variability  also  makes  reducing  the  path  cost  errors  more  challenging  for  the 
participant.  This  implies  that  humans  under  time  pressure  might  be  ill-equipped  to  optimize  cost 
functions,  such  as  the  Time  cost  function,  that  are  very  sensitive  to  small  changes  in  variables. 


168 


Figure  6.38  Cost  differences  for  path  modifications  within  cost  function  (left,  Time;  right, 

Exploration) 


6.6.1.2  Visualizations 

Overall,  visualization  did  not  have  a  main  effect  on  path  planning  performance.  No  significant  main 
effects  were  found  with  path  cost  errors,  time  penalty,  nor  true  time.  Nonetheless,  there  were  some 
trends  based  on  visualization.  With  respect  to  differential  cost,  a  visualization  difference  was  found 
within  the  Exploration  cost  function  only,  where  participants  with  LOEC  visualization  had  smaller 
differential  costs  than  participants  with  both  elevation  contours  and  LOEC  visualization1.  This 
means  that  participants  with  LOEC  had  smaller  differences  between  minimum  and  first  path  costs, 
i.e.,  they  were  close  to  their  optimal  solution  in  their  first  try.  This  implies  that  for  the  LOEC 
visualization  that  aggregated  multiple  variables,  it  was  the  most  effective  for  the  unusual  (and  likely, 
unintuitive  as  it  included  sun  position)  cost  function  of  Exploration.  This  result  is  an  important 
finding  because  it  provides  evidence  that  for  a  multi-variable,  intricate  cost  function,  the  levels  of 
equal  cost  visualization  at  least  helped  participants  start  closer  to  the  average  minimum  cost. 
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With  respect  to  the  amount  of  time  spent  on  optimizing  task,  it  appears  that  participants  that  had 
the  LOEC  visualization  tended  to  equalize  the  amount  of  time  spent  on  the  optimizing  task 
regardless  of  cost  function,  while  participants  that  had  elevations  contours  (either  with  or  without 
LOEC)  spent  most  on  Time  and  Exploration  than  Distance  and  Metabolic.  Another  visualization- 
based  trend  was  detected  in  the  percent  time  spent  modifying  paths.  For  the  Time  cost  function, 
participants  with  the  LOEC  in  their  visualization  spent  significantly  more  of  their  time  modifying 
the  path  than  participants  with  just  the  elevation  contours  visualization.  These  same  elevation 
contours  participants  also  increased  their  time  spent  in  non-optimal  satisficing  for  just  the 
Exploration  cost  function.  Most  importantly,  these  change  in  strategies  based  on  visualization  did 
not  affect  overall  path  planning  performance  as  there  was  no  main  effect  based  on  visualization. 

It  could  be  argued  that  the  difference  in  percent  time  spent  modifying  paths  by  participants  with  the 
LOEC  visualization  did  so  because  the  visualization  is  an  additional,  novel  tool.  However, 
significant  differences  were  only  seen  within  the  Time  cost  function,  and  not  any  other  cost 
function.  This  result  is  significant  when  considering  how  these  very  same  participants  performed 
for  the  Time  cost  function  in  the  off-nominal  case  (discussed  in  the  next  section). 

The  effect  of  visualization  is  most  evident  in  the  Pareto  front  analysis,  where  participants  with  small 
cost  errors  and  short  task  times  were  evaluated.  Under  the  nominal  case,  most  of  the  best 
performers  were  in  the  elevation  contours  group.  Furthermore,  the  participants  in  the  Pareto  front 
set  with  the  smallest  cost  (though  with  longer  times)  were  those  that  had  higher  percent  time  spent 
modifying.  This  suggests  that  manual  sensitivity  analysis,  and  not  just  longer  task  times,  is  essential 
to  optimizing  least-costly  paths.  It  is  important  to  note  though  that  if  the  participant  does  not 
fundamentally  understand  how  to  optimize  the  cost  function,  they  could  spend  large  portions  of 
time  modifying  paths  without  decreasing  the  path  costs.  This  was  seen  within  the  participants  in  the 
Exploration  function  Pareto  front  set.  While  it  was  observed  that  most  of  the  best  performers  were 
in  the  elevation  contours  group,  there  was  no  overall  statistical  main  effect  of  visualization. 


1  This  trend  was  not  seen  for  Metabolic  cost  function. 
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In  summary,  most  of  the  best  performers  (participants  with  low  cost  errors  and  total  time  on  task) 
were  in  the  elevation  contours  group.  With  respect  to  LOEC,  participants  that  had  this  visualization 
tended  to  spend  more  time  modifying  paths  for  the  perceived  most  difficult  function  of  Time,  and 
helped  participants  create  a  first  path  that  was  close  to  optimal  for  the  most  complex  function, 
Exploration.  Surprisingly,  for  the  Metabolic  cost  function,  another  complex  function,  these  trends 
were  not  seen.  Perhaps  the  simple,  shortest  path  heuristic  for  Metabolic  function  circumvented  the 
need  for  participants  to  leverage  the  LOEC  visualization.  Nonetheless,  the  mentioned  trends  are 
indications  that  participants  were  leveraging  the  visualization  to  achieve  least-costly  paths,  although 
exhibited  in  different  manners  for  the  two  cost  functions  of  Time  and  Exploration.  Unfortunately, 
the  visualization  was  not  powerful  enough  to  produce  significant  difference  in  path  planning 
performance  across  all  conditions  (i.e.,  decreased  path  cost  errors  and  total  time  spent  on  task). 


6.6.2  Off-Nominal  Scenario 

Table  6.4  summarizes  the  major  results  for  the  dependent  and  independent  variables  within  Phase  2. 


Table  6.4  Summary  of  results  for  Phase  2 


Independent  Variable  1 

Dependent  Variable 

Visualization 
(EC,  LOEC,  Both) 

Cost  Function 

Scenario 

Path  cost  errors 

No  main  effect 

Time  >  Exploration 

LOEC  Time  nominal  >  off-nominal 
Exploration  nominal  <  off-nominal 

Time  penalty 

No  main  effect 

Time  >  Exploration 

No  main  effect 

Percent  time 
modifying 

No  main  effect 

Interaction  w/  visualization 
EC,  Time  <  Exploration 
LOEC,  Time  >  Exploration 
Both,  Time  ~  Exploration 

No  main  effect 

True  time 

No  main  effect 

Time  >  Exploration 

Nominal  >  Off-nominal 

Differential  cost 

No  main  effect 

Time  >  Exploration 

Time  nominal  >  off-nominal 
Exploration  nominal  ~  off-nominal 
#  of  negative  diff.  costs  nominal  < 
off-nominal 

Non-optimal 
satis-ficing 
(cost  surplus) 

No  main  effect 

Time  >  Exploration 

Nominal  <  Off-nominal 

Non-optimal 
satis-ficing 
(time  surplus) 

No  main  effect 

Time  <  Exploration 

Nominal  <  Off-nominal 
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For  die  off-nominal  cases.  Time  and  Exploration  cost  functions  were  tested.  It  was  hypothesized 
that  path  planning  performance  would  decrease  within  the  off-nominal  scenario,  resulting  in 
participants  introducing  and  increasing  cost  errors  into  their  path  solutions.  Furthermore,  since 
participants  were  explicitly  told  that  they  could  not  depend  on  the  automation  (be  it  the  LOEC  and 
die  displayed  path  costs),  some  evidence  of  distrust  would  emerge  in  their  performance,  such  as 
increased  time  on  task  or  more  time  spent  optimizing  paths.  Based  on  visualization,  it  was 
hypothesized  that  participants  with  elevation  contours  would  benefit  from  their  visualization  as 
opposed  to  those  that  just  had  the  LOEC  visualization.  Thus,  some  automation  bias  was  expected 
for  those  LOEC-only  participants,  in  that  they  would  make  worse  paths  since  they  depended  on  the 
erroneous  visualization. 

6.6.2.1  Effects  of  Degraded  Automation 

Even  though  participants  had  an  idea  of  the  proportion  of  the  cost  function  errors,  without  the 
ability  to  rely  on  the  interface  to  provide  accurate  path  costs,  participants  had  a  poor  sense  of  when 
they  had  reached  a  minimum  path  cost,  introducing  on  average,  an  additional  5  ±  8%  error  in  Time 
and  1.2  ±  1.6%  in  Exploration  cost  functions.  Even  though  these  errors  are  relatively  small,  the 
errors  were  as  high  as  28%  and  5%  for  Time  and  Exploration  functions,  respectively.  In  addition, 
true  times  were  significantly  shorter  in  the  off-nominal  compared  to  the  nominal  case,  which  may  be 
due  to  some  learning  since  the  off-nominal  trials  were  the  last  ones  completed.  There  was  also  a 
significantly  higher  number  of  negative  differential  costs  in  the  off-nominal  condition,  as  compared 
to  the  nominal,  meaning  that  participants  were  submitting  paths  that  were  even  worse  than  the  first 
path  that  was  made.  This  is  important  because  it  indicates  that  there  was  a  decrease  in  path  planning 
performance  under  degraded  automation  conditions.  Participants  did  not  know  they  had  reached  a 
minimum  path  cost,  though  most  were  trying  to  find  it,  as  suggested  by  tire  lack  of  difference  in  time 
penalty  (i.e.,  total  time)  between  scenarios. 

Interestingly,  there  was  no  significant  difference  across  scenarios  for  the  percent  of  time  spent 
modifying  paths.  This  seems  unusual  as  one  might  have  expected  that  the  distmst  in  the  automation 
may  have  led  participants  to  increase  time  spent  modifying,  and  hence,  more  manual  sensitivity 
analysis.  However,  an  increase  in  sensitivity  analysis  was  seen  in  the  metric  of  percent  time  spent 
conducting  non-optimal  satisficing  (i.e.,  participants  attempting  to  find  a  lower  solution  that  was 
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never  achieved).  This  was  an  expected  result  because  participants  would  distrust  the  automation’s 
path  costs,  thus,  they  would  spend  more  of  their  time  searching  for  a  solution.  This  apparent 
disagreement  might  be  due  to  a  shift  in  type  of  sensitivity  analysis.  The  most  frequent  method  of 
conducting  sensitivity  analysis  was  modifying  paths,  the  strategy  chosen  by  26  participants.  The 
other  type  of  sensitivity  analysis,  creating  multiple  paths,  was  only  conducted  by  the  remaining  8 
participant.  The  dependent  measure  of  percent  time  spent  modifying  only  describes  the  first 
strategy. 

Additional  time  on  the  task  overall  might  be  expected  for  the  off-nominal  condition,  yet  there  was 
no  significant  time  penalty  differences  across  scenarios.  If  participants  were  spending  equal  time  on 
both  scenarios,  it  could  be  considered  that  participants  were  not  adding  more  effort  into  finding 
their  path  solutions  or  they  could  have  learned  how  to  optimize  the  paths.  However,  this  is  a  time- 
pressured  task,  and  because  of  it,  participants  may  have  opted  not  to  spend  more  time  in  the 
degraded  automation  condition. 

6.6.2.2  Cost  Functions  &  Visualizations 

Between  cost  functions,  the  results  were  consistent  with  the  nominal  phase  of  the  experiment.  The 
Time  cost  function  was  the  most  difficult  function  as  it  had  significantly  higher  path  cost  errors,  and 
longer  time  penalties  and  true  times  when  compared  to  the  Exploration  cost  function.  However, 
the  assessment  between  scenarios  within  each  cost  function  suggests  that  participants  were  able  to 
adapt  optimization  strategies  for  Time  but  not  the  Exploration  cost  function. 

Under  degraded  automation  conditions,  it  was  expected  that  participants’  performance  would 
deteriorate,  resulting  in  longer  task  times  and  larger  path  cost  errors.  In  terms  of  performance 
across  scenarios,  participants  did  not  spend  more  time  in  the  off-nominal  trials  as  compared  to  the 
nominal  ones.  However,  within  the  Exploration  function,  path  cost  errors  increased  in  the  off- 
nominal  condition  regardless  of  visualization.  Within  the  Time  function,  participants  performed 
differently  within  the  visualization  groups  with  regards  to  path  cost  errors.  Participants  with  the 
LOEC  visualization  decreased  their  Time  path  cost  errors  in  the  off-nominal  condition,  while  there 
was  no  significant  difference  across  scenarios  for  other  two  visualizations. 


173 


It  was  hypothesi2ed  that  since  LOEC  participants  would  exhibit  some  automation  bias  (by 
depending  on  the  erroneous,  automation-generated  visualization  to  make  their  least-costly  paths), 
resulting  in  the  largest  path  cost  errors.  This  was  not  observed  as  there  were  no  visualization 
differences  within  tire  Exploration  function  and  there  was  actually  a  decrease  in  path  cost  errors  for 
the  Time  function. 

The  improved  performance  (decrease  in  path  cost  errors)  of  only  the  LOEC  participants  within  the 
off-nominal  Time  trial  is  surprising,  particularly  in  conjunction  with  the  opposite  effect  for  the 
Exploration  function.  A  similar  trend  (though  not  significant)  was  observed  for  the  participants  that 
had  the  combination  visualization  (both  elevation  contours  and  LOEC).  There  was  no  statistical 
difference  in  Time  path  cost  errors  across  scenarios  for  the  participants  with  elevation  contours 
visualization.  These  differences  in  visualization  were  not  seen  in  the  Exploration  case,  where  all 
participants  performed  the  same. 

There  are  a  few  reasons  that  all  participants  either  did  equally  or  better  in  the  off-nominal  condition 
for  the  Time  cost  function.  First,  this  function  was  conceptually  easier  to  understand  as  it  depended 
on  only  two  variables  (one  of  them  being  slope)  and  hence,  adaptation  was  possible.  There  might 
have  been  a  learning  effect  also.  Noteworthy  are  the  post-questionnaire  comments  of  the  LOEC 
participants  who  mentioned  leveraging  the  path  elevation  profiles1  during  the  off-nominal  trials.  On 
the  other  hand,  the  participants  with  the  elevation  contours  mostly  used  their  visualization  only.  It 
may  be  that  the  more  detailed  information  of  the  path  elevation  profile  was  a  more  useful  aid  for 
determining  changes  in  path  slope. 

Another  reason  for  the  improved  Time  off-nominal  performance  of  just  the  LOEC  participants  may 
be  the  effect  of  manual  sensitivity  analysis  in  the  nominal  phase  of  the  experiment.  These  same 
participants  had  spent  significantly  more  of  their  time  modifying  the  Time  paths,  which  may  have 
led  to  some  learning  and  fundamental  understanding  of  how  to  optimize  this  cost  function. 
Unfortunately,  the  participants  with  both  LOEC  and  elevation  contours  visualizations  did  not 
significantly  reduce  their  path  cost  errors  for  the  Time  off-nominal  trial  (though  there  was  a 


1  See  bottom  of  Figure  6.1. 
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decreasing  trend).  This  may  have  occurred  because  participants  within  this  visualization  group  had 
different  strategies  that  either  favored  LOEC  or  elevation  contours,  as  evidenced  by  the  lack  of 
participant  preference  for  one  visualization  and  post-questionnaires  comments.  By  introducing  two 
simultaneous  visualizations  in  one  group,  the  number  of  strategies  was  increased,  making  it  difficult 
to  compare  effective  strategies  between  the  visualizations. 

Finally,  the  Pareto  front  assessment  revealed  that  the  best  off-nominal  performers  did  not 
exclusively  belong  to  the  elevation  contours  group,  but  more  of  them  were  in  the  LOEC 
visualization  group.  This  is  counter-intuitive  if  it  is  assumed  that  participants  solely  depended  on  the 
visualizations  to  make  their  least-costly  paths.  It  could  be  argued  though  that  performance  under 
degraded  automation  conditions  depended  on  the  amount  of  manual  sensitivity  analysis  (amount  of 
time  spent  modifying  path)  in  the  nominal  condition,  allowing  participants  to  fundamentally 
understand  the  cost  functions  better.  There  was  some  evidence  that  LOEC  participants  did  conduct 
more  path  modifications,  which  in  turn  would  have  assisted  these  participants  in  the  off-nominal 
trials. 


6.7  Conclusions  for  Experiment  2 

In  this  experiment,  four  cost  functions,  three  visualizations,  and  two  scenarios  types  were  examined 
for  their  effect  on  human-led  path  optimization.  The  cost  functions  tested  were  Distance,  Time, 
Metabolic,  and  Exploration;  each  function  depended  on  the  previous,  growing  in  number  of 
variables  and  difficulty.  The  visualizations  included  elevation  contours,  levels  of  equal  cost  (LOEC), 
and  a  combination  of  both  LOEC  and  elevation  contour  lines.  Participants  were  tested  in  both 
nominal  and  degraded  automation  conditions. 

Performance  as  defined  by  errors  and  time  spent  on  task  were  driven  by  the  cost  function,  though 
not  necessarily  the  number  of  variables  manipulated.  The  most  complex  cost  function  (the 
Exploration  cost  function  which  had  the  highest  number  of  variables)  was  not  completely 
understood  by  the  participants.  In  this  case,  the  levels  of  equal  cost  visualization  helped  participants 
initially  make  paths  that  were  close  to  their  optimal.  Participants  without  the  LOEC  visualization 
did  not  know  how  to  fundamentally  optimize  complex  cost  functions,  like  Metabolic  and 
Exploration,  and  hence,  implemented  a  “shortest  path”  heuristics. 
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On  the  other  hand,  the  Time  cost  function  was  the  most  difficult  cost  function  for  participants  to 
optimize  even  though  it  was  relatively  simple,  depending  on  only  two  variables,  Distance  and  slope. 
The  results  indicate  that  if  a  cost  function  is  very  sensitive  to  small  changes  in  variables,  like  the 
Time  function  in  this  experiment,  users  could  have  difficulty  in  optimizing  paths.  These  types  of 
functions  should  be  labeled  as  “overly-sensitive”  cost  functions  in  order  to  differentiate  them  from 
truly  complex  cost  functions,  Exploration.  While  Time  was  an  “overly-sensitive”  cost  function,  it  is 
conceptually  easier  to  comprehend  than  the  Exploration  function.  Thus,  under  degraded 
automation  conditions,  participants  were  able  to  adequately  adapt  path  planning  strategies  for  the 
Time  but  not  the  Exploration  cost  function. 

If  considering  the  two  main  measures  of  performance  in  this  experiment,  path  cost  errors  and  time 
on  task,  the  best  performers  under  the  nominal  conditions  were  mosdy  in  the  elevations  contours 
group.  However,  in  the  off-nominal  cases,  the  LOEC  participants  emerge  as  being  these  better 
performers.  The  reasons  behind  this  may  be  that  while  visualization  did  not  have  a  main  effect  on 
performance,  it  did  influence  the  choice  of  optimizing  strategies.  Participants  with  the  additional 
LOEC  visualization  tended  to  spend  more  time  modifying  the  Time  function  (as  compared  to  users 
that  just  had  elevation  contours)  which  assisted  them  during  degraded  automation  conditions.  The 
levels  of  equal  cost  visualization,  which  aggregates  all  variables  into  one  cost  map,  helped  reduce  the 
complex  problem,  in  terms  of  providing  an  efficient  optimizing  strategy  and  promoted  sensitivity 
analysis  for  difficult  problems. 

In  terms  of  sensitivity  analysis,  participants  could  be  categorized  into  two  types:  manual  and  “whole- 
path”  sensitivity  analysis.  Manual  sensitivity  analysis,  i.e.,  spending  time  modifying  path  solutions, 
was  preferred  by  about  three-fourths  of  the  participants.  Even  though  there  was  no  difference  in 
performance  between  these  two  strategies,  there  was  a  trend  within  the  set  of  best  performers  which 
seemed  to  indicate  that  an  increase  in  manual  sensitivity  analysis  resulted  in  lower  path  cost  errors. 

In  conclusion,  for  the  task  of  path  optimization,  humans  perform  best  when  they  leverage  sensitivity 
analysis.  While  manual  sensitivity  analysis  takes  more  time,  it  allows  users  to  fundamentally 
understand  how  a  path  is  optimized.  As  the  presence  of  the  LOEC  visualization  promoted 
sensitivity  analysis,  this  visualization  is  a  desirable  attribute  in  decision  support  aids  during  the 
optimization  of  paths,  particularly  those  that  depend  on  “overly-sensitive”  cost  functions. 
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7  Meta-Analysis,  Recommendations 


and  Contributions 


7.1  Leveraging  Human-Computer  Collaboration 

The  research  aims  of  dais  research  were  to  investigate  how  to  best  leverage  human-computer 
collaboration  for  the  task  of  path  planning  and  re-planning.  This  was  accomplished  by 
experimentally  testing  a  prototype  path  planner  with  different  degrees  of  automated  data  integration 
across  varying  task  complexities.  In  all  the  experiments,  there  was  a  “what  if’  tool,  which  was 
allowed  participants  to  always  have  the  ability  to  manually  change  the  paths.  In  the  first  experiment, 
automatic  path  generation  and  visualizations  were  assessed  for  their  effect  on  human  path  planning 
performance.  The  three  visualizations  tested  were  1)  elevation  contours,  2)  levels  of  equal  cost 
(LOEC)  visualization,  and  2)  a  combination  of  both.  The  LOEC  visualization  was  an  aggregate  cost 
map.  The  task  complexity  was  increased  through  tire  number  of  manipulated  variables  within  the 
path  cost  function.  In  the  second  experiment,  only  one  automation  architecture  was  examined 
(manual  path  planning).  This  allowed  for  a  better  assessment  of  tire  role  of  sensitivity  analysis  for 
the  task  of  path  planning.  Task  complexity  was  further  increased  by  introducing  a  degraded 
automation  condition  as  well  as  more  intricate  cost  functions.  This  scenario  not  only  made  the  task 
more  difficult,  but  also  mimics  a  condition  where  human  path  planners  must  adapt  to  an  underlying 
incorrect  model  or  a  limited  resolution  map.  Based  on  the  results  of  these  experiments  (described  in 
Chapter  5  and  6),  the  following  about  leveraging  human-computer  collaboration  is  concluded. 

In  the  first  experiment,  passive  (or  manual)  and  active  levels  of  automation  were  compared  across 
three  different  visualizations.  In  terms  of  human-computer  collaboration,  automated  path  segment 
generation  (active  automation)  was  helpful  in  decreasing  the  amount  of  time  spent  on  the  path 
planning  task  by  33%  on  average  and  virtually  eliminated  path  cost  errors.  Unfortunately,  when 
participants  used  the  active  automation,  they  chose  not  to  utilize  the  manual  sensitivity  analysis 
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(“what  if’)  tool  even  though  they  had  done  so  when  manually  creating  paths  (passive  automation). 
This  higher  automation  level  and  the  lack  of  manual  sensitivity  analysis  resulted  in  automation  bias, 
and  the  penalty  was  decreased  situation  awareness,  meaning  participants  were  less  aware  of  the 
elements  and  integration  of  these  than  when  path  planning  with  passive  automation.  Thus,  while 
automated  path  generation  was  beneficial  in  reducing  errors  and  time,  it  was  detrimental  with 
regards  to  situation  awareness.  In  terms  of  visualization,  there  was  no  main  effect  present  in  this 
experiment,  thus,  situation  awareness  was  not  directly  affected  by  the  type  of  visualization.  In  high 
risk  domains,  human  operator  will  have  to  react  to  unexpected  circumstances  and  hence,  must 
depend  on  their  situation  awareness.  Thus,  an  acceptable  trade-off  between  errors  and  situation 
awareness  will  have  decided  upon  for  any  decision  support  system.  Without  the  proper  situation 
awareness,  the  chances  of  reacting  appropriately  in  an  emergency  decrease  due  to  limited 
knowledge-based  reasoning  capabilities. 

In  the  second  experiment  only  one  automation  architecture,  passive  automation,  was  tested  which 
allowed  different  benefits  of  the  visualizations  to  emerge.  While  there  was  no  main  effect  due  to 
visualization,  the  best  performers,  i.e.,  those  that  had  low  path  cost  errors  and  shorter  task  times, 
were  typically  found  in  the  elevation  contours  visualization  group.  This  was  found  only  the  case  for 
the  nominal  cases,  where  participants  could  depend  on  the  automation.  Under  degraded  automation 
conditions,  the  participants  with  the  LOEC  visualization  were  more  prominently  represented  in  the 
set  of  best  performers. 

The  LOEC  visualization  was  also  helpful  for  the  most  complex  cost  function,  Exploration. 
Participants  were  able  to  leverage  the  LOEC  visualization  in  order  to  initially  create  the  Exploration 
least-costly  path.  When  comparing  the  first  and  last  paths  costs  (differential  cost)  for  the  nominal 
Exploration  trial,  participants  with  the  LOEC  were  closer  to  an  optimal  solution  on  the  first 
attempt.  This  would  not  be  important  if  LOEC  participants  performed  poorly,  but  there  was  no 
significant  difference  across  visualizations.  Therefore,  while  the  Exploration  function  was  a 
complex  function,  the  LOEC  visualization  managed  narrow  down  the  problem  space  for 
participants. 

In  terms  of  sensitivity  analysis,  two  types  of  strategies  emerged  among  both  experiments:  manual 
and  “whole-path”  sensitivity  analysis.  In  passive  automation,  both  types  of  sensitivity  analysis  were 
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observed  among  the  participants,  where  25%  of  the  participants  (in  both  experiments)  used  a 
“whole-path”  sensitivity  analysis  strategy.  During  active  automation  (which  was  only  present  in  the 
first  experiment),  most  participants  used  a  “whole-path”  sensitivity  analysis  only.  Thus,  for  this  first 
experiment,  three-fourths  of  participants  switched  from  manual  sensitivity  analysis  in  passive 
automation  to  “whole-path”  in  the  active  automation.  This  switch  was  also  accompanied  by  a 
decrease  in  situation  awareness,  implying  that  manual  sensitivity  analysis  strategy  is  important  for 
development  of  situation  awareness. 

It  is  important  to  address  the  difference  between  “whole-path”  sensitivity  analysis  that  occurred 
within  active  as  opposed  to  passive  automation.  In  active  automation,  whole  paths  were  created 
after  one  waypoint  was  defined  by  the  participant.  In  passive  automation,  whole  paths  were  created 
after  the  participant  defined  all  the  waypoints.  In  essence,  the  type  of  sensitivity  analysis  was  the 
same,  however,  in  the  passive,  the  user  took  a  more  important  role  in  generating  the  path.  Since 
situation  awareness  decreased  for  the  active  automation,  the  implication  is  that  whole-path 
sensitivity  analysis,  where  tire  human  operator  is  more  involved  in  the  generation  of  path  solutions, 
is  more  beneficial,  or  desirable,  than  one  where  the  automation  defines  most  of  the  path. 

Within  both  experiments,  there  was  no  statistical  difference  between  the  sensitivity  analysis  types 
with  respect  to  path  cost  errors.  However,  within  the  second  experiment,  there  are  some  trends  that 
seem  to  imply  that  there  are  benefits  to  conducting  manual  sensitivity  analysis.  Thus,  it  is 
hypothesized  that  human  operators  that  conduct  manual  sensitivity  analysis  during  path 
optimization  are  better  able  to  adapt  their  optimization  strategy  to  accommodate  degraded 
automation  condition.  This  was  most  prominent  within  tire  LOEC  participants1.  These  participants 
tended  to  spend  large  portions  of  optimizing  time  conducting  manual  sensitivity  analysis  (i.e.,  larger 
percent  time  spent  modifying)  under  nominal  conditions.  In  conjunction,  they  performed  better  in 
the  off-nominal  conditions  when  compared  to  other  participants.  Specifically,  LOEC  participants 
decreased  the  path  cost  errors  in  the  Time  off-nominal  path  while  the  others  showed  no  significant 
improvement.  Additionally,  LOEC  participants  were  more  present  among  tire  best  performers  in 


1  LOEC  participants  in  the  second  experiment,  as  degraded  automation  conditions  were  only  tested 
in  this  experiment. 
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the  off-nominal  trials.  These  results  also  imply  that  the  presentation  of  the  LOEC  visualization 
promotes  manual  sensitivity  analysis. 

The  question  then  becomes  “Are  there  benefits  directly  related  to  the  type  of  sensitivity  analysis 
strategy?”  To  address  this  question,  path  planning  performance  between  participants  with  different 
sensitivity  analysis  strategies  were  compared.  Of  the  two  main  performance  metrics  (path  cost 
errors  and  task  times),  cost  error  was  selected  as  the  appropriate  evaluation  metric  because  task  time 
could  bias  the  comparison  since  conducting  path  modifications  is  a  more  time-consuming  strategy 
than  creating  new  paths.  In  both  experiments,  there  was  no  statistical  difference  with  respect  to 
path  cost  errors  between  the  participants  that  chose  manual  waypoint  as  opposed  to  whole-path 
sensitivity  analysis.  However,  in  the  Pareto  front  analysis  for  the  second  experiment  (which  included 
participants  with  both  types  of  sensitivity  analysis  strategies),  there  was  an  apparent  trade-off 
between  path  error  and  time.  In  general,  an  increase  in  task  time  led  to  a  decrease  in  path  cost 
errors,  which  was  accompanied  by  an  increase  in  percent  time  spent  modifying  path,  as  exemplified 
by  the  Pareto  front  participants  for  the  nominal  Time  cost  function  (see  section  6.5.1  and  Figure 
6.35). 

This  trend  of  improved  path  costs  with  an  increase  in  path  modifications  prompted  a  meta-analysis 
that  included  participants  in  both  experiments  for  all  cost  functions1.  Within  each  cost  function, 
participants  were  given  a  performance  score  based  on  the  path  cost  errors  (1  through  3,  representing 
best,  average,  and  worst  performance)  and  ranked  by  the  percent  amount  of  time  spent  modifying 
paths  (see  Appendix  C  for  details).  Since  it  was  already  shown  that  no  difference  in  path  cost  errors 
could  be  detected  between  sensitivity  analysis  types2,  the  meta-analysis  focused  on  only  the 
participants  that  conducted  manual  sensitivity  analysis.  There  was  a  significant  moderate  to  low 
correlation  between  the  performance  score  and  amount  of  path  modifications  across  both 
experiments  (Spearman’s  rho  =  -0.19,  p  =  0.008).  These  results  imply  that  more  path  modifications 


1  Only  for  the  passive  automation  condition  from  first  experiment  as  only  this  automation 
architecture  was  tested  in  the  second  experiment. 

2  Additionally,  there  was  no  significant  correlation  between  performance  score  and  type  of  sensitivity 
analysis. 
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result  in  lower  path  cost  errors.  Thus,  while  conducting  “whole-path”  sensitivity  analysis  did  not 
result  in  higher  path  cost  errors  when  compared  to  manual  sensitivity  analysis,  there  was  an 
increasing  benefit  to  conducting  more  path  modifications.  The  limitation  to  this  benefit  is  the 
additional  time  it  requires  to  conduct  manual  sensitivity  analysis.  Under  time  pressure,  “whole-path” 
sensitivity  analysis  participants  on  average  spent  1.4  minutes  optimizing,  while  the  other  strategy 
took  about  twice  as  long. 

While  manual  sensitivity  analysis  helped  participants  reduce  path  cost  errors,  the  fact  remains  there 
were  some  cost  functions  that  participants  were  poor  at  optimizing,  namely  the  Time  cost  function 
and  the  Sun  Score  function.  While  it  may  be  hypothesized  that  large  Sun  Score  errors  were  due  to 
participants  not  understanding  how  to  optimize  this  function,  the  same  can  not  assumed  for  the 
Time  cost  function.  The  Time  cost  function  was  highly  sensitive  to  small  path  modifications.  In 
other  words,  path  changes  resulted  in  large  cost  differences.  To  a  certain  extent,  this  was  also 
observed  for  the  Sun  Score.  Future  investigation  may  want  to  consider  how  to  further  leverage 
active  automation  for  “overly-sensitive”  cost  functions,  such  as  developing  a  hybrid  automated  path 
generation  where  users  defined  many  waypoints  but  the  automation  calculates  the  least  costly  path 
segments  between  these  human-designated  waypoints 1 . 

In  summary,  the  key  to  a  successful  path  planning  and  re-planning  decision  support  system  is  to 
appropriately  balance  human  and  computer  role  allocation,  such  that  humans  conduct  enough 
sensitivity  analysis  to  maintain  situation  awareness.  Manual  sensitivity  analysis  is  a  time-consuming 
task,  which  might  be  appropriate  during  planning,  but  this  strategy  will  have  to  be  limited  under  re¬ 
planning  time  pressures.  Based  on  this  thesis  research,  knowledge-based  reasoning  is  supported  by 
manual  sensitivity  analysis,  which  was  lacking  when  path  were  automatically  generated  for  users. 
While  leveraging  automatic  path  generation  results  in  better  path  performance  (i.e.,  lower  cost  errors 
and  shorter  task  times),  the  dependence  on  this  aid  leads  to  automation  bias  and  decreased  situation 
awareness.  The  missing  components  at  the  higher  level  of  automation  were  time  spent  on  the  task 
and  manual  sensitivity  analysis  of  the  paths.  Therefore,  human  operators  should  use  aids  that 
promote  sensitivity  analysis  during  the  planning  phase  in  order  to  best  understand  the  relationship 


1  Currently,  in  the  passive  automation,  the  path  segments  between  waypoints  is  a  straight  line. 
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between  variables  and  path  cost.  Then,  when  confronted  with  the  time  pressured  task  of  re¬ 
planning,  he/ she  may  leverage  the  knowledge  gained  from  having  already  conducted  the  sensitivity 
analysis. 

With  respect  to  visualizations,  the  elevation  contours  representation  promoted  low  cost  paths  with 
short  task  time  when  human  operators  could  depend  on  the  automation.  However,  the  levels  of 
equal  cost  (LOEC)  visualization,  which  aggregated  total  path  costs,  helped  participants  initially 
create  least-costly  paths  for  the  most  complex  cost  function.  Furthermore,  LOEC  visualizations 
appear  to  promote  manual  sensitivity  analysis  in  the  nominal  case,  which  in  turn  was  beneficial  in 
the  degraded  automation  condition,  where  low  path  errors  and  time  was  observed  for  participants 
with  this  visualization.  Finally,  two  types  of  sensitivity  analysis  strategies  were  observed,  one  that 
leveraged  the  available  “what  if’  tools  for  waypoint  modification  and  the  other  that  created  whole 
paths.  While  there  was  no  difference  in  performance  across  the  strategies  (i.e.,  both  sensitivity 
analysis  strategies  were  equally  effective),  the  increased  use  of  the  manual  waypoint  sensitivity 
analysis  (i.e.,  path  modifications)  led  to  decreased  path  cost  errors. 

7.2  Design  Recommendations 

7.2.1  Path  Planning  and  Re-Planning  Decision  Support  Aids 

Based  on  the  work  domain  analysis  and  the  analysis  of  the  experiments  conducted,  the  following 
design  recommendations  for  decision  support  aids  that  support  both  planning  and  re-planning  of 
paths  are  suggested.  These  recommendations  are  generalized  beyond  space  application  (i.e.,  real¬ 
time  aids  for  human  planetary  exploration),  and  can  be  applied  to  any  complex  decision  support 
systems  that  assists  operators  in  tire  tasks  of  path  planning  and  re-planning  within  other  domains, 
such  as  UAV  operations,  search  and  rescue  robots,  and  ground  soldiers.  The  recommendations 
specific  to  planetary  EVA  are  discussed  in  a  subsequent  section. 

The  results  of  the  experiments  conducted  in  this  thesis  indicate  a  positive  relationship  between 
manual  sensitivity  analysis  with  both  situation  awareness  and  performance  under  degraded 
automation  conditions.  Thus,  the  first  design  recommendation  is: 


182 


•  Provide  “what  if”  tools  for  path  planning  and  re-planning  in  order  to  encourage 
sensitivity  analysis.  A  “what  if’  tool  was  essential  for  manual  sensitivity  analysis  and  thus, 
such  a  tool  within  an  automated  aid  will  help  operators  find  alternative  routes  and  plans 
when  faced  with  re-planning.  Sensitivity  analysis,  for  both  planning  and  re-planning,  should 
be  encouraged  as  it  was  shown  to  possibly  lead  to  lower  path  costs.  In  the  PATH  test  bed, 
the  basic  “what  if’  tool  included  moving,  adding,  and  deleting  of  waypoints,  and  path  cost 
changes  were  only  visible  once  a  modification  was  executed.  Further  manual  sensitivity 
analysis  tools  may  be  beneficial  (see  also  7.3). 

With  respect  to  visualization,  there  was  no  main  effect  due  to  this  test  condition.  However,  there 
are  visualization  recommendations  based  on  the  observed  trends  for  the  experiments  conducted: 

•  Make  aggregate  cost  information  available  to  human  operator  through  direct 
perception  visualizations.  With  the  levels  of  equal  cost  visualization  (LOEC),  human 
operators  are  leveraging  direct  perception,  which  “pushes”  a  complex  problem  solving  task 
to  a  lower  level  of  cognitive  control.  The  LOEC  visualization,  which  aggregates  multivariate 
data,  was  particularly  useful  for  complex  cost  functions  and  helped  to  visualize  high  cost 
areas.  Hence,  the  path  planner  may  want  to  take  advantage  of  this  visualization  under  time 
pressure  to  create  new  paths.  Additionally,  LOEC  users  tended  to  conduct  more  sensitivity 
analysis.  Thus,  if  providing  an  aid  like  the  LOEC  visualization  during  pre-planning  and 
training,  the  human  operator  will  likely  develop  more  accurate  mental  models  of  how  the 
cost  functions  interact  with  important  variables  (for  instance,  terrain  characteristics  for  the 
planetary  EVA  application),  allowing  for  better  adaptation  to  imperfect  automation. 

•  Make  raw  (or  basic)  information  available  to  the  human  operator.  When  the  decision 
support  aid’s  automation  models  are  incomplete  (e.g.,  low  map  fidelity),  tire  operator  needs 
to  have  access  to  the  most  basic  of  data  (which  is  domain  specific)  to  adapt  their  new  paths 
and  accommodate  for  discrepancies.  In  this  thesis  investigation,  the  raw  data  was  terrain 
models,  and  best  performance  under  nominal  conditions  were  observed  with  this  basic 
information  visualization  (elevation  contours).  Providing  access  and  tire  choice  to  both  basic 
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and  aggregate  information  is  important  to  support  both  nominal  and  off-nominal  path 
planning  conditions,  however,  overlaying  them  was  not  beneficial. 

The  work  domain  analysis  (WDA)  conducted  for  this  research  examined  planetary  EVAs.  However, 
there  are  some  broader  implications  for  this  analysis  as  it  was  one  that  included  and  assessed  why 
and  how  paths  were  re-planned.  Hence,  the  WDA,  in  conjunction  with  the  results  of  the 
experiment,  suggest  the  following  design  recommendations: 

•  A  human  operator  should  be  involved  in  both  the  path  planning  and  execution 
phases  of  the  path.  In  order  for  re-planning  to  be  effective,  the  human  operator  should  be 
knowledgeable  about  how  the  path  was  planned.  He/ she  will  then  be  familiar  with  the  cost 
functions  and  reasoning  behind  the  path  selection.  This  may  be  accomplished  in  the 
planning  phase,  where  operators  may  leisurely  conduct  an  extensive  manual  sensitivity 
analysis  of  path  solutions.  If  the  human  agent  has  a  more  complete  understanding  of  the 
plan,  upon  path  execution  (e.g.,  actually  traversing  or  operating  some  robotic  agent  along 
that  path),  he/ she  will  be  more  aware  of  deviations  from  the  plan.  Furthermore, 
understanding  how  deviations  occurred,  e.g.,  identifying  erroneous  cost  functions,  is 
essential  for  re-planning  new  path  solutions. 

•  The  path  planning  aid  should  be  flexible.  Though  not  directly  tested  in  the  PATH 
experiments,  the  work  domain  analysis  indicated  that  re-planning  revolves  around  assessing 
the  different  options  available.  This  can  only  be  accomplished  by  providing  a  flexible  aid  for 
path  planners.  In  general,  this  corresponds  to  providing  the  human  operator  the  ability  to 
add  new  or  remove  constraints  (e.g.,  physical  terrain  obstacles)  and  modify  cost  function 
models.  Specifically  for  the  domain  of  planetary  exploration,  other  automated  function 
include  changing  the  cost  function  models  based  on  type  of  transportation  modality  (e.g., 
walking  vs.  driving)  and  reprioritization  of  sites  or  waypoints  along  the  path. 

The  above  design  recommendations  can  be  applied  to  a  variety  of  domains  that  require  human 
operators  to  path  plan  and  re-plan,  including  transportation  domains.  In  these  circumstances,  the 
domain  is  complex,  requiring  the  operator  to  manipulate  multiple  variables,  sometimes  having  to  re¬ 
plan  under  time  pressure.  Therefore,  the  same  lessons  learned  from  this  investigation  (e.g.,  the 
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importance  of  sensitivity  analysis,  the  use  of  display  visualizations,  and  the  need  for  flexibility  of 
automated  aid)  may  help  operators  in  their  routing  tasks  and  in  dealing  with  unexpected  future 
events  or  imperfect  automation. 

In  terms  of  the  most  common,  commercial  path  planning  aids,  such  as  automobile  GPS  and  on-line 
path  planners,  these  recommendations  are  still  relevant  but  should  be  interpreted  more  broadly. 
Most  on-line  path  planners  find  shortest  or  fastest  routes  between  two  locations,  and  provide  users 
some  ability  to  constrain  paths,  like  avoiding  tolls  or  highways.  The  more  sophisticated  planners 
allow  users  to  organize  trips  with  multiple  intermediate  waypoints,  while  the  planner  automatically 
finds  routes  in  between.  Automobile  planners  (and  up-and-coming  mobile  GPS  systems)  have 
become  highly  automatic,  planning  routes  and  directing  drivers  to  their  destination.  Like  on-line 
planners,  paths  are  optimized  based  on  time  or  distance.  Real-time  updates  allow  car  navigation 
systems  to  revise  paths,  avoiding  traffic  and  leading  users  to  the  cheapest  gas  station  or  the  closest 
rest  stop.  Additionally,  theses  systems  can  find  alternative  paths  and  re-route  drivers  if  they  deviate 
from  path  (or  can  be  triggered  by  user). 

Unfortunately,  neither  of  these  planners,  on-line  or  automobile,  permit  users  to  modify  the  actual 
path  itself  (i.e.,  adjust  multiple  paths  segments).  In  addition,  if  automobile  planners  follow  the  on¬ 
line  trend,  they  will  start  multi-waypoint  path  planning,  and  hence,  increase  the  amount  of  user 
interactivity.  Both  of  these  system  could  introduce  improvements  at  the  planning  phase,  particularly 
giving  users  the  ability  to  modify  paths.  Integrated  “what  if’  tools  (similar  to  those  developed  for 
PATH)  would  help  users  create  paths  that  suit  their  preferences  and  assess  the  cost  (in  terms  of 
distance  and  time)  of  their  modifications.  Furthermore,  visualizations  (analogous  to  LOEC)  could 
assist  users  in  the  path  planning  phase  by  highlighting  areas  which  users  prefer,  like  scenic  areas,  or 
want  to  avoid,  like  construction  zones.  During  the  planning  phase,  users  may  become  familiar  with 
the  pros  and  cons  of  particular  routes,  saving  some  and  selecting  one.  If  re-planning  is  required,  the 
user  could  revert  to  one  of  the  previously  created  paths  that  better  suits  his/her  needs.  Thus,  while 
the  set  of  design  recommendations  apply  more  directly  to  complex,  time-pressured  domains,  beyond 
planetary  EVAs,  they  can  be  considered  in  a  broader  context  and  lessons  learned  are  relevant  to 
automated  planners  used  daily. 
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7.2.2  Future  Real-time  Planetary  Decision  Support  Aids 


The  investigation  of  this  thesis  focused  on  understanding  how  human-automation  collaboration  can 
be  leveraged  for  the  geospatial  problem  solving  task  of  path  planning.  Such  an  automated  path 
planner  would  only  be  part  of  a  larger,  more  sophisticated  computer  tool  for  future  astronauts 
exploring  the  Moon  or  Mars.  In  addition  to  the  already  discussed  design  recommendations,  the 
following  requirements  are  proposed  specifically  for  real-time  planetary  decision  support  aids. 

•  Functional  requirements: 

o  The  real-time  decision  support  aid  should  assist  astronauts  or  mission  controllers  in 
determining  if  re-planning  is  needed.  Thus,  the  aid  must  present  to  the  human 
operator  actual  and  expected  path  costs  along  the  traversal. 

o  Real-time  tracking  of  the  position  of  the  astronaut  and  other  costs,  such  as  energy 
consumed,  is  required.  Position  (and  potentially  orientation)  tracking  should  be 
overlaid  with  the  planned  path,  though  this  depends  on  the  navigation  system  in 
place  on  the  planetary  surface. 

o  Automatic  assessment  of  variables  that  are  responsible  for  deviations  from  planned 
path.  For  instance,  if  the  source  of  errors  is  terrain-specific,  path  costs  deviations 
can  be  correlated  to  terrain  characteristics  in  order  to  determine  the  true  relationship 
between  cost  and  terrain  (i.e.,  correct  cost  model). 

•  Information  requirements: 

o  A  complete  real-time  decision  support  aid  should  have  access  to  all  the  inputs  and 
constraints  listed  in  the  Planetary  EVA  Framework  (Chapter  3).  These  inputs  and 
constraints  do  not  need  to  be  presented  to  the  astronaut  or  mission  controller  at  all 
times.  However,  human  operators  must  have  access  to  this  information  if  it  is 
required  for  path  re-planning. 
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o  The  decision  support  aid  should  have  a  terrain  map  (both  an  aerial  map  and  a  digital 
elevation  model,  i.e.,  elevation  information)  with  an  overlay  of  the  planned  EVA. 

o  The  pertinent  associated  path  costs  must  be  included  with  the  path  itself.  The  costs 
(already  identified  in  the  Chapter  3)  include  for  instance,  energy  consumed.  The 
EVA  schedule  is  also  an  essential  to  display  alongside  the  path. 

o  Path  re-planning  is  likely  to  be  triggered  by  incomplete  cost  function  models  or  poor 
map  resolution.  Providing  the  astronaut  with  a  sense  of  the  level  of  uncertainty 
would  be  beneficial  (e.g.,  map  resolution  or  confidence  intervals  of  path  costs). 

1.2.2.X  Hardware  Implications 

The  information  needed  for  path  planning  and  re-planning  can  be  displayed  on  any  type  of 
computer  screen.  There  are  several  options  that  have  been  previously  suggested  for  use  by 
astronauts  during  EVAs:  heads-up  display  (HUD),  a  spacesuit-imbedded  screen,  or  a  separate 
computer  display  (Figure  7.1).  HUDs  and  wearable  computers  are  feasible  but  may  interfere  with 
the  astronaut’s  field  of  view  (already  encumbered  with  wearing  a  helmet).  Hence,  a  screen  imbedded 
or  attached  to  the  spacesuit  arm  or  a  small,  portable  computer  is  another  option. 


Figure  7.1  Future  NASA  concepts  of  information  systems.  From  left  to  right:  heads-up 
displays  (image  s99_04197),  spacesuit  imbedded  screen  (image  jsc2004el8850),  separate 

computer  screen  (image  jsc2004el8859). 
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Above  all,  astronauts  (or  mission  controllers)  must  be  able  to  interact  with  the  map  and  re-planning 
aid,  and  be  able  to  conduct  a  sensitivity  analysis  on  planned  paths  in  order  to  decrease  path  costs 
and  maintain  situation  awareness,  critical  for  knowledge-based  reasoning.  However,  these 
requirements  have  associated  hardware  implications  with  respect  to  input  devices.  High  levels  of 
human-computer  interaction  for  re-planning  make  buttons  and  voice  commands  inadequate  as  they 
are  poor  methods  for  communicating  geospatial  information.  Thus,  the  physical  implementation  of 
re-planning  decision  support  aids  requires  input  devices  akin  to  a  computer  mouse  (i.e.,  pointing, 
clicking,  and  dragging  on  a  screen).  Unfortunately,  current  pressurized  spacesuit  technology  limits 
hand  dexterity,  which  restricts  the  use  of  the  traditional  computer  mouse.  If  tire  decision  aid  were 
on  a  portable  computer,  a  track  ball  could  be  used  for  pointing  and  dragging  (with  the  addition  of  a 
button  used  with  the  other  hand).  If  an  imbedded  screen  were  to  be  implemented  (such  as  in  Figure 
7.1),  a  trackball  is  unlikely  to  be  used.  Alternatively,  it  could  be  feasible  to  integrate  a  stylus-type 
device  on  to  the  tip  of  a  gloved  finger  to  point  and  drag  on  the  touch  screen;  a  button  could  be 
incorporated  on  the  other  hand  or  finger.  This  would  also  promote  direct  manipulation  of 
information  on  a  screen,  be  it  on  a  portable  computer  or  on  the  spacesuit  arm. 

7.3  Possible  Future  Research 

The  following  is  a  list  of  potential  improvements  for  PATH  and  possible  new  research  directions 
based  on  the  experimental  results  of  this  thesis. 

•  PATH  improvements:  PATH  was  purposefully  designed  to  be  a  relatively  simple  path 
planner  (in  order  to  conduct  controlled  experimentation).  The  improvements  mentioned 
below  are  based  on  observation  and  include  the  comments  from  participants. 

o  Normalize  path  costs  relative  to  one  path.  This  would  help  users  compare  costs 
between  paths  more  easily  and  consequently,  the  bar  cost  graph  would  depict  the 
path  cost  differences.  Potentially,  this  may  increase  the  user’s  ability  to  differentiate 
between  small  path  cost  differences. 


188 


o  Provide  to  the  user  a  sense  of  how  much  each  variables  or  cost  functions  is 
contributing  to  the  overall  path  cost.  This  can  be  accomplished  simply  through  a  pie 
chart  or  stacked  bar  graph  which  reflected  the  percent  each  variable  contributes. 

o  Provide  larger  map  and/ or  zooming  capability  for  the  map.  This  improvement  may 
affect  the  total  task  time  (as  it  is  an  additional  functionality  that  users  may  abuse) 
while  not  affecting  path  cost  errors,  yet  it  may  provide  the  user  the  ability  to  make 
smaller  resolution  changes.  Similarly,  path  cost  and  elevation  profiles  may  be 
enlarged. 

o  Prevent  or  assist  users  from  making  paths  that  cross  obstacles.  Participants  were  not 
permitted  to  make  paths  that  included  an  obstacle,  and  thus  were  required  to  restart 
die  paths  if  they  mistakenly  intersected  one.  The  planner  could  automatically  fix  the 
user’s  path  and  avoid  obstacles. 

o  Provide  additional  “what  if’  tools.  Other  aids  that  may  assist  in  sensitivity 
analysis  are:  1)  an  “undo”  modification  button,  which  would  allow  users  to  return  to 
previous  path,  2)  a  “copy”  path  button,  3)  a  “save”  path  button  and  4)  a  log  and 
upload  capability  of  saved  paths  with  corresponding  costs.  These  improvements  are 
relatively  simple  compared  to  the  often  requested  functionality  of  real-time  updating 
of  path  cost  changes  while  simultaneously  modifying  path.  This  dynamic  version  of 
path  modifications  could  potentially  facilitate  manual  sensitivity  analysis. 

•  LOEC  improvements  and  research  opportunities: 

o  Change  color  gradient  used  for  LOEC  visualization.  The  yellow-purple  color 
combination  was  selected  in  PATH’S  design  because  it  did  not  overlap  with  tire  path 
colors  while  still  providing  the  range  of  colors  for  the  user  to  differentiate  between. 
The  color  differences  can  be  made  more  apparent  to  the  user  by  implementing  a 
non-linear  color  gradient  (currently,  it  is  linear).  This  would  have  the  effect  of 
malting  relatively  high  cost  areas  more  purple.  If  the  path  color  constraint  is 
removed,  there  are  other  methods  of  improving  tire  color  differences,  such  as 
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choosing  more  colors  (for  instance,  a  rainbow  set)  and  providing  the  user  with  a 
dynamic  slider  (i.e.,  making  the  user  select  the  set  of  colors  he/ she  was  to  visualize). 
These  changes  would  need  to  be  tested  against  the  present  LOEC  visualization  in 
order  to  determine  if  these  improve  upon  the  current  design. 

o  Make  LOEC  visualization  a  dynamic  aid.  Currently,  the  LOEC  visualization  is  static 
as  there  is  only  one  goal.  However,  a  dynamic  version  of  the  LOEC  visualization 
may  be  implemented.  One  way  to  do  this  is  by  permitting  the  user  to  drag  the  goal 
location  on  the  map  while  the  automation  calculates  and  presents  the  new  LOEC 
(for  the  new  goal),  giving  the  user  the  ability  to  assess  relative  costs  between 
intermediate  waypoints  and  allow  him/her  to  observe  how  the  LOEC  visualization 
changes  as  it  is  moved  around  the  map.  This  would  be  particularly  useful  if  users 
were  tasked  to  create  paths  that  traverse  many  critical  way-areas.  The  second  way  to 
implement  a  dynamic  LOEC  visualization  is  by  giving  the  user  the  ability  to  change 
the  cost  function  (variables  or  relationships)  and  observe  how  the  visualization  itself 
changes  based  on  the  cost  function  modifications.  Additionally,  by  making  LOEC  a 
dynamic  visualization,  it  could  be  considered  a  type  of  sensitivity  analysis  aid.  Such 
an  improvement  on  the  LOEC  visualization  would  be  computationally  expensive, 
and  thus  some  preliminary  studies  should  assess  both  the  potential  advantages  over  a 
static  LOEC,  implications  of  modifying  cost  functions  models  on  overall  path 
planning  performance,  and  the  ease  of  use  of  such  an  aid. 

•  Trade-off  between  active  automation  and  situation  awareness:  This  thesis  investigated 
one  implementation  of  active  automation,  where  the  user  selected  only  one  waypoint 
between  a  given  start  &  goal  and  the  automation  plotted  the  least  costly,  in-between  path 
segments.  Other  active  automation  implementations  (more  appropriately  named  hybrid 
automation)  could  require  users  to  select  multiple  waypoints  whereas  the  automation  fills  the 
least  costly  path  segments  between  these.  This  would  increase  the  amount  of  the  path  that  is 
user  defined.  Future  research  may  examine  if  this  method  of  leveraging  automation 
encourages  users  to  conduct  more  manual  sensitivity  analysis  and  to  maintain  situation 
awareness  while  keeping  path  cost  errors  small  and  short  task  times. 
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•  Utilizing  PATH  as  a  strategic  planner:  Presently,  PATH  is  designed  as  a  tactical  planner, 
meaning  it  is  to  be  used  for  planning  paths  based  on  current,  pre-determined  conditions 
(e.g.,  start,  goal,  and  sun  position).  A  strategic  planner  would  imply  that  these  conditions 
could  be  set  and  changed  to  reflect  future  scenarios.  A  potential  new  research  direction  with 
PATH  could  be  to  assess  the  effectiveness  of  PATH  as  a  strategic  planner,  tire  required 
“what  if’  tools  to  facilitate  this  type  of  planning,  and  to  further  understand  the  operator’s 
capability  to  conduct  these  types  of  searches. 

•  Modify  PATH  for  real-time  path  re-planning:  Even  though  PATH  is  a  planning  aid,  it 
can  be  modified  to  conduct  real-time  path  re-planning.  This  could  be  accomplished  by 
simulating  events  that  require  the  user’s  to  re-plan.  Furthermore,  PATH  could  be  adapted 
for  Earth  conditions  (e.g.,  local  terrain  map)  and  used  in  real-time  to  re-plan  traverses. 

7.4  Contributions 

The  aim  of  this  research  was  to  enable  knowledge-based  reasoning  for  geo-spatial  path  planning 
problem  through  both  visualization  and  human-automation  interaction.  In  order  to  explore  this 
problem,  different  human-automation  role  allocations  were  examined  and  tested  in  order  to 
understand  how  humans  conduct  complex  optimizations  under  different  automated  assistance  levels 
in  a  geospatial  task.  As  a  result,  this  thesis  begins  to  fill  in  a  critical  interdisciplinary  research  gap 
between  human  interaction  and  artificial  intelligence  by  examining  how  automation  impacts  path 
planning  performance.  While  focused  on  human  planetary  exploration,  in  part  due  to  the  recent 
national  attention  on  space  exploration  and  humans  returning  to  the  Moon,  the  acquired  knowledge 
for  human  interaction  with  automation  planners  applies  to  a  many  other  domains  such  as  air  traffic 
control,  robotic  exploration,  and  rescue  on,  above,  and  below  the  Earth.  Furthermore,  the  results  of 
this  thesis  could  be  of  particular  interest  to  industry  as  automated  planners  are  being  integrated  in  a 
myriad  of  commercially-available  technologies. 

This  thesis  has  provided  the  following  contributions: 

•  A  framework  for  human-robotic  planetary  EVA  planning,  including  key  input  variables, 
constraints,  and  outputs  in  the  form  of  information  requirements  (Chapter  3).  This 
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framework  is  broad  enough  that  it  can  be  adapted  to  other  human-robotic  planning 
domains. 

•  A  prototype  path  planner  based  on  lunar  terrain  models  and  planetary  cost  functions 
(Chapter  4).  While  this  planner  does  not  incorporate  all  the  Planetary  EVA  Framework 
elements,  it  focuses  on  fundamental  pieces,  the  planned  path,  different  visualizations,  and 
increasingly  complex  cost  functions.  Thus,  it  can  be  the  basis  of  a  more  complete,  future 
planner. 

•  Quantification  of  path  planning  performance  across  different  decision  support 
visualizations,  cost  function  complexity,  automated  path  generation,  and  degraded 
automation  condition  (Chapters  5  and  6).  It  is  concluded  that  sensitivity  analysis  is  key  to 
maintaining  situation  awareness  and  improving  path  planning  performance,  though  the 
benefits  are  limited  to  the  amount  of  time  available  to  conduct  path  modifications.  The 
levels  of  equal  cost  visualization  did  not  have  a  direct  effect  on  performance,  but  it  may 
encourage  sensitivity  analysis,  which  was  necessary  for  adequate  performance  for  users  under 
the  degraded  automation  condition. 

•  Identification  of  cognitive  strategies  people  use  when  interacting  with  automated  path 
planners,  which  has  not  been  addressed  in  previous  research. 

•  Design  recommendations  for  future  real-time  decisions  support  aids  for  planetary  EVAs, 
including  information,  functional,  and  hardware  implications. 

Planetary  exploration  is  a  reality  with  rovers  on  Mars.  These  robotic  agents  are  the  eyes  and  legs  of 
the  scientists  on  Earth.  If  we  are  to  send  humans  back  to  the  Moon  in  just  over  a  decade,  and  move 
on  to  Mars  from  there,  we  need  to  focus  attention  towards  the  technologies  supporting  people  on 
other  planetary  surfaces  as  these  technologies  maintain  humans’  safety  in  these  extreme 
environments.  Human-system  integration  is  the  key  to  the  success  of  any  mission  that  includes  both 
humans  and  robotic  agents,  and  this  research  concentrated  on  developing  robust  decision  support 
systems  for  EVAs  to  ensure  safe  and  productive  missions. 
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Appendix  A:  Supplementary  Information 
for  Experiment  1 


Pilot  Experiment  Implications 

Before  the  first  experiment  was  undertaken,  a  pilot  experiment  was  conducted.  The  pilot  revealed 
several  issues  in  the  experimental  methods  that  were  subsequently  addressed.  Participants  were 
divided  into  one  of  three  groups:  1)  manual  path  planning  with  an  elevation  contours  map,  2) 
manual  path  planning  with  the  option  of  using  the  levels  of  equal  cost  (LOEC)  visualization  and 
elevation  contours  maps,  and  3)  automatic  path  planner  with  elevation  contours  map  (highest  level 
of  automation).  Since  participants  in  the  second  group  were  allowed  to  change  the  map 
visualization  they  were  provided,  they  were  able  to  opt-out  from  using  the  LOEC  visualization. 
This  problem  was  fixed  by  eliminating  the  option  to  choose  map  visualizations. 

Additionally,  participants  in  the  third  group  only  had  to  request  the  automatic  path  planner  for  the 
least-costly  path,  resulting  in  experimental  bias;  the  path  planning  task  was  too  easy  for  these 
participants.  This  problem  was  fixed  in  the  first  experiment  by  changing  how  the  automatic  path 
planner  functioned.  Instead  of  providing  a  least-costly  path  from  the  given  start  and  goal  locations, 
the  user  determines  an  intermediate  waypoint  and  the  automatic  planner  only  finds  least-costly  paths 
from  the  start  to  the  intermediate  waypoint  and  from  there,  to  the  goal.  The  user’s  waypoint  is 
constrained  to  be  within  a  given  critical  way-area.  The  critical  way-areas  were  designed  to  always 
include  the  least-costly  path  from  the  given  start  and  goal  locations. 

An  analysis  of  cost  function  presentation  order  (i.e.,  which  cost  function  was  seen  first)  revealed  a 
learning  effect,  and  that  counterbalancing  did  not  work  as  expected.  The  effect  detected  was  that 
participants  who  did  the  hard  and  then  easy  cost  functions  performed  better  than  the  participants 
who  did  the  easy  and  then  hard  cost  functions.  Thus,  in  order  to  equalize  the  learning  effect,  only 
one  order  is  implemented.  Finally,  the  selected  trials  varied  in  difficulty,  which  prevented  accurate 
performance  comparisons.  More  emphasis  was  thus  placed  on  selecting  trials  to  ensure  equal 
difficulty  across  conditions. 


Pre-Questionnaire 


A  brief  pre-questionnaire  was  given  to  the  participants  before  they  started  the  experiment.  The 
information  gathered  was  used  to  balance  the  visualization  groups  and  to  test  (and  control)  for 
participant  experience  which  may  have  affected  path  planning  performance. 


193 


Subject  number: _  Group  number: _ 

Age: _  Gender:  male/female  Color  blindness:  yes/no 

Occupation:  graduate  student/ undergraduate/ staff/ faculty/ unaffiliated 
Major: _ 

How  often  do  you  play  video  games? 

Never/Less  than  1  hour  per  week/Between  1  and  4  hours  per  week/ 
Between  1  and  2  hours  per  day/More  than  2  hours  per  day. 

Please  mark  the  skills  you  feel  proficient  in: 

Orienting  a  map/Triangulating  position/Estimating  distance  on  map/ 
Interpreting  topological  map/ Off-trial  hiking. 


Visualization  Description 


Each  subject  was  given  the  description  of  the  visualization  he/ she  would  be  using  during  the 
experiment.  Subjects  read  only  the  descriptions  (shown  below)  for  their  own  particular 
visualization,  and  no  further  instructions  were  provided. 

Description  given  to  subjects  for  the  elevation  contours  visualization: 


Start:  (32,186)  Goal:  (174,177)  Coordinates:  (  ,  ) 


•  Your  map  has  the  following 
visualization: 

-  The  black  objects  are 
obstacles,  and  you  cannot  go 
through  them. 

-  The  grey  filled  contours  is  the 
elevation  contour  map.  They 
show  levels  of  (relatively)  equal 
elevation. 
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Description  given  to  subjects  for  the  levels  of  equal  cost  (LOEC)  visualization: 


Your  map  has  the  following 
visualization: 

-  The  black  objects  are  obstacles,  and 
you  cannot  go  through  them. 

-  The  color  gradient  represents  levels  of 
equal  cost  (LOEC).  This  is  a 
visualization  of  how  the  automation 
finds  a  path  of  minimum  cost. 

-  The  automation  calculates  the 
minimum  cost  (based  on  the  objective 
function)  from  every  point  in  the  map 
to  the  goal.  We  have  taken  that 
minimum  cost  and  displayed  it  as  a 
color.  The  result  is  a  color  gradient 
that  indicates  directions  of  minimum 
cost.  Yellow  indicates  a  relatively 
cheap  path  is  possible  while  purple 
indicates  that  only  a  more  expensive 
path  to  the  goal  is  possible. 


Description  given  to  subjects  for  the  LOEC  and  elevation  contours  visualization: 


Group  3 


Your  map  has  the  following 
visualization: 

-  The  black  objects  are  obstacles, 

and  you  cannot  go  through  them. 

-  The  line  contours  is  the  elevation 
contour  map.  They  show  levels  of 
relatively  equal  elevation. 

-  The  color  gradient  represents  levels 
of  equal  cost  (LOEC).  This  is  a 
visualization  of  how  the  automation 
finds  a  path  of  minimum  cost. 

-  The  automation  calculates  the 
minimum  cost  (based  on  the 
objective  function)  from  every  point 
in  the  map  to  the  goal.  We  have 
taken  that  minimum  cost  and 
displayed  it  as  a  color.  The  result 

is  a  color  gradient  that  indicates 
directions  of  minimum  cost.  Yellow 
indicates  a  relatively  cheap  path  is 
possible  while  purple  indicates  that 
only  a  more  expensive  path  to  the 
goal  is  possible. 


Situation  Awareness  Questions 

The  situation  awareness  (SA)  questions  were  designed  to  address  SA  levels  1  and  2.  SA  level  1 
focuses  on  the  perception  of  elements,  while  level  2  is  about  the  integration  and  comprehension  of 
these  elements.  Therefore,  the  SA  questions  (asked  after  each  trial)  specifically  test  participants  on 
their  perception  of  elements  about  the  previously  tested  trial,  such  as  sun  position  or  obstacles  that 
had  to  be  avoided,  and  their  integration  and  comprehension  about  how  these  elements  affect  path 
costs.  The  SA  questions  are  shown  below  as  presented  to  the  participants,  and  include  the 
corresponding  SA  level  the  question  targets. 
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Practice  Question  (SA  level  1) 


Practice  Question 


•  The  end  goal  state  is: 
a  South  of  start 
b  West  of  start 
c)  North  of  start 
d  I  don't  know 


Trial  1:  Question  1  (SA  level  1)  &  Question  2  (SA  level  2) 


Trial  1,  Question  1 


Trial  1,  Question  2 


For  the  last  trial,  what  was  the 
sun’s  elevation  angle? 


Elevation  Angle:  4S° 


For  the  last  trial,  what  is  the  relative  elevation  difference  between 
the  start  and  the  goal? 

The  start  is  higher  than  the  goal, 
b'  The  goal  is  higher  than  the  start. 

The  start  and  the  goal  are  about  the  same  elevation, 
d  I  don't  know 


Trial  2:  Question  1  (SA  level  1)  &  Question  2  (SA  level  2) 


Trial  2,  Question  1 


Trial  2,  Question  2 


What  function  was  this  path  evaluated  with? 

♦  “  A*.  “ 

SS  -(cos(20)  ♦  2Wcos<2*)  -  2) 


0  “  0^r  - 

ES-(cos<2*)+2) 


Assume  the  same  elevation  map  fot 
both  examples  (i.e.,  start  elevation  for 
top  example  is  same  as  start  elevation 
in  bottom  example;  same  for  goal). 
Both  of  these  paths  are  the  least 
costly  path  from  start  to  goal  that 
goes  through  the  given  way  area, 
using  the  sun  score.  What  is  the 
reason  as  to  why  they  look  so 
different? 


c. 

AS  *  cos(20)  +  2 

1  don't  know 


The  sun  position  of  the  top  path  is 
westward  but  the  bottom  path  has  a 
southward  sun  position. 

V 

The  sun  position  of  the  top  path  is 
southward  but  the  bottom  path  has  a 
westward  sun  position. 

A 

The  sun  position  of  the  top  path  ts 
westward  but  the  bottom  path  has  a 

p  t 

eastward  sun  position. 

‘  •  c 

di  1  don't  know 
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Trial  3:  Question  1  (SA  level  1)  &  Question  2  (SA  level  2) 

Trial  3,  Question  1 


Trial  3,  Question  2 


For  the  last  trial,  what  was  the 
sun's  elevation  and  azimuth 
angle? 

Azimuth  angle  -  90®  (North), 
elevation  angle  -  45" 


Azimuth  angle  -  90°  (North), 
elevation  angle  -  30° 


1  ilMOanto#*  W 


Azimuth  angle  -  0®  (East), 
elevation  angle  -  30° 


ill  I  don't  know 


1 


‘Vi/ 


,  k  %. 


This  is  the  least  costly  path  from  start 
to  goal  that  goes  through  the  given 
way  area,  using  the  elevation  score.  If 
the  sun  had  been  coming  from  the 
north  rather  than  the  east,  the  new 
optimal  path  would  have  been: 

More  costly  than  the  path  shown  here 
Less  costly  than  the  path  shown  here 
Would  be  the  same  cost  as  the  path 
shown  here 
I  don't  know 


Trial  4:  Question  1  (SA  level  1)  &  Question  2  (SA  level  2) 


Trial  4,  Question  1 


Trial  4,  Question  2 
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Appendix  B:  Supplementary  Information 
for  Experiment  2 


Visualization  Description 

Similar  to  the  first  experiment,  each  subject  was  given  the  description  of  the  visualization  he/ she 
would  be  using  during  the  experiment.  Subjects  read  only  the  descriptions  (shown  below)  for  their 
own  particular  visualization,  and  no  further  instructions  were  provided.  These  descriptions  are  more 
detailed  than  those  in  the  first  experiment  because  participants  had  commented  (during  the  de-brief) 
that  they  would  have  preferred  more  explanation  about  the  visualization  (particularly  those 
participants  with  the  levels  of  equal  cost  visualization). 


Description  given  to  subjects  for  elevation  contours  visualization: 


Your  Visualization 


Black  objects  are  obstacles 
(terrain  slope  >  15°). 
Elevation  contour  lines  in 
gray  gradient 

-  White  -  highest 
elevation 

-  Dark  gray  -  lower 
elevation 


More  about  visualization 


Use  the  visualization! 
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Description  given  to  subjects  for  the  levels  of  equal  cost  (LOEC)  visualization: 


Your  Visualization 


Black  objects  are  obstacles 
(terrain  slope  >  15°). 

Color  gradient  is  the  levels 
of  equal  cost  (LOEC) 
visualization. 

What  the  colors  mean: 

-  The  smallest  possible  cost 
from  that  point  in  the  map  to 
the  goal 

-  Yellow  indicates  a  relatively 
cheap  cost  to  the  goal  is 
possible  while  purple 
indicates  a  relatively  more 
expensive  cost  to  the  goal  is 
possible. 

LOEC  does  not  show  you 
the  path,  just  the  minimum 
cost. 


How  do  I  use  LOEC? 


Use  the  visualization! 


These  two  points  are  about 
the  same  color,  thus  there 
exists  a  path  from  here  to 
the  goal  that  are  of  equal 
(minimum)  cost 


.  The  point  on  the  right  is 
I  more  purple,  thus  its  min 
cost  from  the  goal  is 
more  expensive  than  the 
point  on  the  left 


These  are  the  directions 
of  minimum  cost  for  this  fc 
particular  example.  ** 


How  do  I  use  LOEC? 


Use  the  visualization! 


How  do  I  use  LOEC? 


Use  the  visualization! 


This  path  is  not 
parallel  to  a  mlnii 
cost  direction 


|  This  path  Is  more 
1  likely  to  be  cheaper 
because  it  has 
segments  that  are 
parallel  to  a  min  cost 
direction 


Now  that  you  know  what 
directions  are  most 
advantageous  for  min 
cost,  it  desirable  for  your 
path  (or  segments  of)  to 
be  parallel  to  these  min 
directions 


* 


The  right  point  has  a 
more  expensive  min  cost 
as  compared  to  the  point 
on  the  left 


v' 


V 

* 


-  • 


LOEC  also  lets  you  see  how 
I  the  cost  function  interacts 
with  the  terrain. 


K 

\ 


*  • 

LOEC  can  help  you 
avoid  areas  that  may 
incur  higher  costs 


How  do  I  use  LOEC? 


How  can  you  see  what  the  j 
different  terrain  elevations  are?  I 


Use  the  visualization! 


LOEC  also  lets  you  see  how 
the  cost  function  interacts 
with  the  terrain 


vj 
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Description  given  to  subjects  for  LOEC  and  elevation  contours  visualization: 


Your  Visualization 


Use  the  visualization! 


Black  objects  are  obstacles 
(terrain  slope  >  15°). 

Color  gradient  is  the  levels 
of  equal  cost  (LOEC) 
visualization. 

What  the  colors  mean: 

-  The  smallest  possible  cost 
from  that  point  in  the  map  to 
the  goal 

-  Yellow  indicates  a  relatively 
cheap  cost  to  the  goal  is 
possible  while  purple 
indicates  a  relatively  more 
expensive  cost  to  the  goal  is 
possible 

LOEC  does  not  show  you 
the  path,  just  the  minimum 
cost. 


How  do  I  use  LOEC? 


Use  the  visualization! 


These  two  points  are  about 
the  same  color,  thus  there 
exists  a  path  from  here  to 
the  goal  that  are  of  equal 
(minimum)  cost 


.  The  point  on  the  right  is 
I  more  purple,  thus  its  min 
cost  from  the  goal  is 
I  more  expensive  than  the 
point  on  the  left 


mmm 

These  are  the  directions 
of  minimum  cost  for  this 
particular  example.  H 


L 


How  do  I  use  LOEC? 


Use  the  visualization! 


a  \ 


— 


[  This  path  is  not 
parallel  to  a  minimum  II 

* 


f’L. 

l  Now  that  you 


I  This  path  is  more 
*  likely  to  be  cheaper 
because  it  has 
segments  that  are 
parallel  to  a  min  cost 
direction 


Now  that  you  know  what 
directions  are  most 
advantageous  for  min 
cost,  it  desirable  for  your 
path  (or  segments  of)  to 
be  parallel  to  these  min 
directions 


How  do  I  use  LOEC? 


The  right  point  has  a 
more  expensive  min  cost 
as  compared  to  the  point 
on  the  left 


v' 


yj. 


Use  the  visualization! 


LOEC  also  lets  you  see  how 
the  cost  function  interacts 
I  with  the  terrain. 


I  LOEC  can  help  you 
x  avoid  areas  that  may 
JB  incur  higher  costs 

v  ^ 


How  do  I  use  LOEC? 


I - x  '  * 

r  '  '  ' 

How  can  you  see  what  the  f  a 

different  terrain  elevations  are?  ay 


Use  the  visualization! 


.  LOEC  also  lets  you  see  how 
I  the  cost  function  interacts 
I  with  the  terrain. 


tj 
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Assessment  of  Covariates 


Before  the  experiment  was  conducted,  each  participant  completed  a  map  planning  test  (with 
corresponding  score)  and  self-rated  experience  with  hiking  and  map-use.  These  two  metrics  were 
correlated  to  the  two  main  dependent  measures,  path  cost  errors  and  time  penalty,  in  order  to 
determine  if  they  were  significant  covariates. 

Map  planning  test  scores  were  tested  for  correlation  to  path  cost  error  and  time  penalty  for  each  of 
the  6  tested  conditions  (4  nominal,  2  off-nominal  trials).  There  were  no  significant  correlations 
between  scores  and  path  cost  errors,  or  with  time  penalties.  Even  though  visualization  groups  were 
not  balanced  for  map  score,  there  was  no  significant  difference  between  the  groups  with  respect  to 
scores  (Kruskal-Wallis  test,  %2(2,  N  =  34)  =  0.42,  p  =  0.81).  Thus,  scores  were  not  used  as  a 
covariate  in  the  analysis. 


Spearman’s  rho  correlation  results  between  map  planning  test  scores  and  path  cost  error 


Map  planning  test  score 

Nominal  Scenarios 

Off-nominal  Scenario  1 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Correlation  coefficient 

0.11 

-0.14 

0.15 

-0.01 

0.04 

0.06 

p-value  (2-tailed) 

0.54 

0.43 

0.40 

0.96 

0.81 

0.73 

Spearman’s  rho  correlation  results  between  map  planning  test  scores  and  time  penalty 


Map  planning  test  score 

Nominal  Scenarios 

Off-nominal  Scenario  1 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Correlation  coefficient 

-0.15 

-0.03 

-0.16 

-0.18 

-0.10 

-0.20 

p-value  (2-tailed) 

0.40 

0.87 

0.36 

0.31 

0.57 

0.27 

The  original  map  &  hiking  experience  5-point  scale  permitted  participants  to  self-rate  how  much 
familiarity  they  had  with  hiking  and/or  using  topographic  maps  in  their  traverses.  Only  one 
participant  self-rated  as  having  neither  hiking  nor  map  use  experience.  In  order  to  correlate 
experience  with  path  cost  errors  and  time  penalty,  the  categories  were  rearranged  to  four  bins:  1) 
little/no  map  experience,  2)  some  hiking  experience,  3)  intermediate  experience  hiking  with  maps, 
and  4)  expert  hiking  with  maps.  The  visualization  groups  were  not  balanced  for  experience  and  no 
significant  difference  between  visualization  groups  was  detected  (Kruskal-Wallis  test,  %2(2 ,  N  =  34) 
=  0.52,  p  =  0.77). 


With  respect  to  experience  and  performance  correlations,  there  were  no  significant  correlations 
between  the  experience  and  path  cost  errors,  or  with  time  penalties.  There  was  only  a  marginally 
significant  correlation  between  time  penalty  and  experience  within  the  Distance  cost  function  trial. 
For  all  the  other  conditions,  this  trend  was  not  significant  (and  smaller  in  effect  size).  Thus, 
experience  was  not  used  as  a  covariate  in  the  analysis. 
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Spearman’s  rho  correlation  results  between  map  &  hike  experience  and  path  cost  error 


Map  &  hike  experience 

Nominal  Scenarios 

Off-nominal  Scenario  1 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Correlation  coefficient 

0.03 

-0.12 

-0.13 

-0.12 

0.25 

0.08 

p-value  (2-tailed) 

0.86 

0.49 

0.48 

0.51 

0.15 

0.66 

Spearman’s  rho  correlation  results  between  map  &  hike  experience  and  time  penalty 


Map  &  hike  experience 

Nominal  Scenarios 

Off-nominal  Scenario  1 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Correlation  coefficient 

-0.30 

-0.16 

-0.08 

0.01 

-0.21 

-0.07 

p-value  (2-tailed) 

0.08 

0.39 

0.65 

0.97 

0.24 

0.72 

Path  Cost  Errors 


Descriptive  statistics  for  path  cost  errors  within  visualizations 


Visualization 

Nominal 

Off-nominal 

Group 

Distance 

Time 

Metabolic 

Time 

Elevation  contours 

1.04  ±  0.04 

1.54  ±0.17 

1.07  ±  0.02 

1.05  ±  0.01 

1.58  ±  0.22 

1.11  ±  0.03 

LOEC 

1.03  ±  0.03 

1.63  ±  0.14 

1.08  ±  0.03 

1.07  ±  0.02 

1.47  ±  0.16 

1.10  ±  0.03 

Both  visualizations 

1.02  ±  0.02 

1.64  ±  0.10 

1.06  ±  0.02 

1.06  ±  0.02 

1.53  ±  0.20 

1.10  ±  0.04 

Total 

1.03  ±  0.03 

1.60  ±  0.14 

1.07  ±  0.02 

1.06  ±  0.02 

1.53  ±  0.20 

1.10  ±  0.03 

•  In  Phase  1,  there  was  a  significant  difference  in  path  cost  errors  between  cost  functions 
(Friedman  test:  %2(3,  N  =  32)  =  72.56,  p  <  0.001). 


Wilcoxon  Sign  tests  results  for  comparisons  between  cost  functions,  Phase  1 


NOMINAL 

Time 

Metabolic 

Exploration 

Distance 

Z  =  -5.01,  p  <  0.0001 

Z  =-4.11,  p  <0.0001 

Z  =-3.99,  p  <0.0001 

Time 

Z  =-5.01,  p  <0.0001 

Z  =-5.09,  p  <0.0001 

Metabolic 

Z  =  -0.74,  p  =  0.46 
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Kruskal- Wallis  test  results  for  path  cost  errors  differences  between  visualization  groups 
within  cost  function 


Cost  Function 

Scenario 

Kruskal-Wallis,  x2 

p-value 

Distance 

Nominal 

X2(2,33)  =  0.67 

p  =  0.72 

Time 

Nominal 

X2(2,34)  =  2.25 

p  =  0.33 

Metabolic 

Nominal 

X2(2,33)  =  2.33 

p  =  0.31 

Exploration 

Nominal 

X2(2,34)  =  3.94 

p  =  0.14 

Time 

Off-nominal 

X2(2,34)  =  1.83 

p  =  0.40 

Exploration 

Off-nominal 

X2(2,34)  =  0.06 

p  =  0.97 

Wilcoxon  Sign  test  results  for  path  cost  errors  differences  between  cost  functions 


Cost  Function  Comparisons 

Wilcoxon  Sign  test  &  p-value 

Regardless  of 
visualization 

Elevation 

contours 

LOEC 

Both 

visualizations 

Time  off-nominal  —  Exploration  off-nominal 

Z  =  -5.09, 

p  <  0.0001 

Time  nominal  —  Time  off-nominal 

Z  =  -1.96, 
p  =  0.05 

Z  =  -0.39, 
p  =  0.70 

Z  =  -2.22, 

p  =  0.026 

Z  =  -1.60, 

p  =  0.11 

Exploration  nominal  —  Exploration  off-nominal 

Z  =  -4.86, 

p  <  0.0001 

Z  =  -3.06, 

p  =  0.002 

Z  =  -2.93, 
p  =  0.003 

Z  =  -2.58, 

p  =  0.01 

Total  Time:  Time  Penalty 


Descriptive  statistics  for  time  penalty 


Visualization 

Nominal 

Off-nominal  1 

Group 

Distance 

Time 

Metabolic 

ISlSS^jSBISSllB 

Time 

Elevation  contours 

1.69  ±  0.30 

2.16  ±  0.93 

1.78  ±  0.42 

2.01  ±  0.62 

2.00  ±  0.77 

1.84  ±  0.55 

LOEC 

2.06  ±  0.53 

2.28  ±  0.80 

2.07  ±  0.56 

1.89  ±  0.49 

2.38  ±  0.94 

1.84  ±  0.64 

Both  visualizations 

1.84  ±  0.40 

2.49  ±  1.16 

1.93  ±  0.70 

2.08  ±  0.69 

2.17  ±  0.66 

2.12  ±  0.80 

Total 

1.82  ±  0.40 

2.30  ±  0.95 

1.94  ±  0.57 

1.99  ±  0.59 

2.18  ±  0.79 

1.93  ±  0.66 

Repeated  analysis  of  variance  for  Phase  1 


Factor 

F-value 

p-value 

Cost  function 

F(1.91,  57.34)  =  7.84 

p  =  0.001 

Visualizations 

F(2,30)  =  0.36 

p  =  0.701 

Cost  function  x  Visualization 

F(3.82,  57.34)  =  1.36 

p  =  0.262 
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Pair-wise  comparisons,  Bonferroni  tests  (regardless  of  visualization) 


NOMINAL3 

Time 

Metabolic 

Exploration 

Distance 

t(32)  = -3.18,  p  =  0.003 

t(33)  =  -0.63,  p  =  0.53 

t(33)  =  -1.19,  p  =  0.24 

Time 

t(32)  =  -3.89,  p  <  0.0001 

t(32)  =  -2.89,  p  =  0.007 

Metabolic 

t(33)  =  -1.04,  p  =  0.31 

a.  Bonferroni  adjusted  p-value  —  0.008 


Simple  contrast  of  Time  function 


NOMINAL 

Distance 

Metabolic 

Exploration 

Time 

F(l,  30)  =  10.46,  p  =  0.003 

F(l,  30)  =  15.13,  p  =  0.001 

F(l,  30)  =  8.44,  p  =  0.007 

•  Simple  main  effects  within  LOEC  visualization  showed  that  there  was  no  effect  due  to  cost 
function  (F(3,  28)  =  1.85,  p  =  0.16).  However,  a  single  comparison  between  Exploration 
and  Time  cost  functions  showed  significantly  less  time  was  spent  on  Exploration  than  the 
Time  cost  function  (pair-wise  comparison  with  least  significant  difference,  p  =  0.036). 


Repeated  analysis  of  variance  for  Phase  2 


Factor 

F-value 

p-value 

Cost  function 

F(l,  29)  =  17.33 

p  <  0.0001 

Visualizations 

F(2,  29)  =  0.13 

p  =  0.88 

Scenario 

F(l,  29)  =  0.34 

p  =  0.56 

Cost  function  x  Visualization 

F(2,  29)  =  1.65 

p  =  0.21 

Scenario  x  Visualization 

F(2,  29)  =  0.39 

p  =  0.68 

Function  x  Scenario 

F(l,  29)  =  0.072 

p  =  0.79 

Function  x  Scenario  x  Visualization 

F(2,  29)  =  0.23 

p  =  0.80 

Percent  Time  Spent  Modifying  Path 


Descriptive  statistics  for  percent  time  spent  modifying  path 


Visualization 

Group 

Nominal 

Off-Nominal 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Elevation 

Contours 

29.25%  ± 
22.76 

24.09%  ± 
25.10 

32.95%  ± 
24.84 

34.44%  ± 
27.74 

26.56%  ± 
30.25 

32.62%  ± 
26.39 

LOEC 

43.28%  ± 
14.10 

44.33%  ± 
17.61 

39.53%  ± 
24.81 

38.94%  ± 
25.31 

41.35%  ± 
25.31 

30.11%  ± 
26.56 

Both 

visualizations 

37.23%  ± 
23.83 

42.39%  ± 
20.26 

35.18%  ± 
27.29 

48.74%  ± 
28.47 

39.11%  ± 
26.70 

40.24%  ± 
25.80 

Total 

36.37%  ± 
20.97 

36.56%  ± 
22.72 

35.80%  ± 
25.01 

40.52%  ± 
27.07 

35.41%  ± 
27.57 

34.27%  ± 
25.81 
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Repeated  analysis  of  variance  for  Phase  1 


Factor 

F-value 

p -value 

Cost  function 

F(3,  93)  =  0.788 

p  =  0.504 

Visualizations 

F(2,  31)  =  1.125 

p  =  0.337 

Cost  function  x  Visualization 

F(6,  93)  =  1.433 

p  =  0.210 

Main  effect  due  to  LOEC  visualization  for  Phase  1 


Factor 

F-value 

p -value 

Cost  function 

F(3,96)  =  0.86 

p  =  0.47 

LOEC 

F(1 ,32)  =  2.32 

p  =  0.14 

Cost  function  x  Visualization 

F(3,96)  =  1.45 

p  =  0.23 

•  Within  the  Time  cost  function,  LOEC  groups  (both  groups  with  LOEC  visualization)  were 
compared  with  elevation  contours  group  using  a  simple  effect  test  of  Time  (F(l,32)  =  6.52,  p 
=  0.016);  LOEC  participants  spent  more  percent  time  modifying  the  Time  cost  function 
than  the  other  visualization  group. 

Simple  main  effect  due  to  cost  function 


Visualization 

F-value 

p-value 

Elevations  contours 

F(3,  29)  =  1.30 

p  =  0.29 

LOEC 

F(3,  29)  =  0.32 

p  =  0.81 

Elevations  contours  &  LOEC 

F(3,  29)  =2.15 

p  =  0.12 

Point-wise  comparisons  within  Elevation  contours  group 


NOMINAL 

Metabolic 

Exploration 

Time 

t(ll)  = -2.30,  p  =  0.042 

t(ll)  =  -2.70,  p  =  0.021 

•  Point-wise  comparison  between  Exploration  and  Metabolic  cost  function  within  the 
visualization  with  both  elevation  contours  &  LOEC  was  significant  (t(10)  =  -2.34,  p  =  0.041) 


Point-wise  comparisons  for  Exploration  cost  function  (Least  significant  difference  test) 


NOMINAL 

LOEC 

Elevation  Contours  & 
LOEC 

Elevation  Contours 

p  =  0.70 

p  =  0.22 

LOEC 

p  =  0.41 
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Repeated  analysis  of  variance  for  Phase  2 


Factor 

F-value 

p -value 

Cost  function 

F(l,  31)  =  0.25 

p  =  0.62 

Visualizations 

F(2,  31)  =  1.04 

p  =  0.37 

Scenario 

F(l,  31)  =2.02 

p  =  0.165 

Cost  function  x  Visualization 

F(2,  31)  =  4.10 

p  =  0.026 

Scenario  x  Visualization 

F(2,31)  =  0.61 

p  =  0.55 

Function  x  Scenario 

F(l,31)  =  1.54 

p  =  0.23 

Function  x  Scenario  x  Visualization 

F(2,31)  =  0.012 

p  =  0.99 

Simple  main  effect  due  to  cost  function  for  Phase  2 


Visualization 

F-value 

p-value 

Elevations  contours 

F(l,  31)  =4.01 

p  =  0.054 

LOEC 

F(l,  31)  =3.77 

p  =  0.061 

Elevations  contours  &  LOEC 

F(l,  31)  =0.76 

p  =  0.39 

True  Time 


Descriptive  statistics  for  true  time 


Visualization 

Nominal 

Off-nominal  1 

Group 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Elevation 

89.07  ± 

135.12  ± 

94.38  ± 

96.57  ± 

96.37  ± 

56.08  ± 

Contours 

35.13 

87.54 

66.78 

72.65 

78.42 

47.93 

LOEC 

129.24  ± 

155.38  ± 

97.49  ± 

103.62  ± 

141.69  ± 

91.64  ± 

66.36 

82.18 

52.58 

61.87 

107.77 

81.26 

Both 

98.41  ± 

178.61  ± 

99.99  ± 

110.89  ± 

138.48  ± 

103.09  ± 

visualizations 

61.16 

110.68 

62.49 

41.09 

90.41 

66.41 

Total 

105.09  ± 

155.75  ± 

97.20  ± 

103.49  ± 

124.66  ± 

82.80  ± 

56.41 

92.96 

59.28 

58.88 

92.18 

67.28 

Repeated  analysis  of  variance  for  Phase  1 


Factor 

F-value 

p-value 

Cost  function 

F(3,  93)  =  7.70 

p  <  0.0001 

Visualizations 

F(2,  31)  =  0.53 

p  =  0.59 

Cost  function  x  Visualization 

F(6,  93)  =  0.62 

p  =  0.71 
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Simple  contrasts 


NOMINAL 

Distance 

Metabolic 

Exploration 

Time 

F(l,  31)  =  10.88,  p  =  0.002 

F(l,  31)  =  15.66,  p<  0.0001 

F(l,  31)  =  11.61,  p  =  0.002 

Repeated  analysis  of  variance  for  Phase  2 


Factor 

F-value 

p -value 

Cost  function 

F(l,  31)  =  24.15 

p  <  0.0001 

Visualizations 

F(2,  31)  =  1.23 

p  =  0.31 

Scenario 

F(1 ,31)  =  4.62 

p  =  0.040 

Cost  function  x  Visualization 

F(2,  31)  =  0.17 

p  =  0.84 

Scenario  x  Visualization 

F(2,  31)  =  0.44 

p  =  0.65 

Function  x  Scenario 

F(l,  31)  =  0.26 

p  =  0.62 

Function  x  Scenario  x  Visualization 

F(2,  31)  =  0.26 

p  =  0.77 

Differential  Cost 

Descriptive  statistics  for  differential  cost 


Visualization 

Nominal 

r 

Off-nominal  [ 

Group 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Elevation 

Contours 

9.98  ±  9.17 

23.56  ±  18.58 

7.48  ±  9.59 

4.16  ±  4.11 

4.84  ±  15.50 

0.13  ±  1.91 

LOEC 

5.02  ±2.48 

30.63  ±  24.26 

10.38  ±  7.17 

1.34  ±  1.65 

16.68  ±  24.25 

3.18  ±  3.30 

Both 

visualizations 

7.78  ±  6.36 

25.76  ±  15.77 

8.72  ±  5.90 

3.32  ±  2.17 

14.94  ±  30.12 

1.44  ±  7.01 

Total 

7.66  ±  6.82 

26.56  ±  19.44 

8.82  ±  7.63 

2.97  ±  3.05 

11.94  ±  23.71 

1.54  ±  4.59 

Distribution  of  participants  with  zero  differential  cost  between  visualizations,  Phase  1 


Cost  Function 

Pearson  yj-  test 

p-value 

Distance 

X2  (2,34)  =  1.02 

p  =  0.60 

Time 

X2  (2,34)  =  0.43 

p  =  0.81 

Metabolic 

X2  (2,34)  =  0.47 

p  =  0.79 

Exploration 

X2  (2,34)  =  3.02 

p  =  0.22 
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Kruskal- Wallis  test  results  for  differential  cost  across  visualizations 


Cost  Function 

Scenario 

Kruskal-Wallis,  x2 

p-value 

Distance 

Nominal 

X2(2,34)  =  1.00 

p  =  0.61 

Time 

Nominal 

X2(2,34)  =  0.53 

p  =  0.77 

Metabolic 

Nominal 

X2(2,34)  =  0.76 

p  =  0.68 

Exploration 

Nominal 

X2(2,34)  =  4.99 

p  =  0.082 

Time 

Off-nominal 

X2(2,34)  =  2.50 

p  =  0.29 

Exploration 

Off-nominal 

X2(2,34)  =  4.06 

p  =  0.13 

Visualization  comparisons  between  within  Exploration  cost  function  (Mann-Whitney  tests) 


EXPLORATION 

NOMINAL 

LOEC 

Elevation  Contours  & 
LOEC 

Elevation  Contours 

Z  = -1.54,  p  =  0.12 

Z  =  -0.19,  p  =  0.85 

LOEC 

Z  =  -2.33,  p  =  0.02 

In  Phase  1,  there  was  a  significant  difference  in  differential  costs  between  cost  functions 
(Friedman  test:  %2(3,  N  =  34)  =  41.37,  p  <  0.0001). 


Wilcoxon  Sign  test  results  of  pair-wise  comparisons  for  differential  cost  between  cost 
functions,  Phase  1 


Cost  Function  Comparisons 

Wilcoxon  Sign  test 
&  p-value 

Result 

Distance  -  Time 

Z  =  -4.42,  p  <  0.0001 

Distance  <  Time 

Distance  -  Metabolic 

Z  =  -0.83,  p  =  0.41a 

Distance  £  Metabolic 

Distance  —  Exploration 

Z  =  -3.58,  p  <  0.0001b.c 

Exploration  <  Distance 

Time  -  Metabolic 

Z  =  -4.33,  p  <  0.0001 

Metabolic  <  Time 

Time  —  Exploration 

Z  =  -4.83,  p  <  0.0001 

Exploration  <  Time 

Metabolic  —  Exploration 

Z  =  -3.61,  p  <  0.0001 

Exploration  <  Metabolic 

a.  Only  within  the  LOEC  visualization  group:  Z  =  -2.05,  p  =  0.041 

b.  Only  within  elevation  contours  group:  Z  =  -1.65,  p  =  0.099 

c.  Only  within  elevation  contours  &  LOEC  group:  Z  =  -1.87,  p  =  0.062 


Distribution  of  participants  with  negative  differential  cost  between  visualizations,  Phase  2 


Cost  Function 

Pearson  x2  test 

p-value 

Time 

X2(2,34)  =  1.00 

p  =  0.61 

Exploration 

X2(2,34)  =  2.55 

p  =  0.28 
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Distribution  of  participants  with  zero  differential  cost  between  visualizations,  Phase  2 


Cost  Function 

Pearson  Jj  test 

p-value 

Time 

X2( 2,34)  =  1.00 

p  =  0.61 

Exploration 

X2(2,34)  =  1.98 

p  =  0.37 

Distribution  of  participants  with  negative  and  zero  differential  cost  between  visualizations, 
Phase  2 


Cost  Function 

Pearson  x2  test 

p-value 

Time 

X2(4,34)  =  2.55 

p  =  0.64 

Exploration 

X2(4,34)  =  5.23 

p  =  0.27 

Wilcoxon  Sign  test  results  for  differential  costs  between  cost  functions 


Cost  Function  Comparisons 

Wilcoxon  Sign  test  &  p-value 

Regardless  of 
visualization 

Elevation 

contours 

LOEC 

Both 

visualizations 

Time  off-nominal  —  Exploration  off-nominal 

-IS 

IS 

mm 

9H 

Time  nominal  —  Time  off-nominal 

Z  =  -2.67, 

p  =  0.008 

Z  =  -1.87, 

p  =  0.062 

Z  =  -1.42, 

p  =  0.16 

Z  =  -1.25, 

p  =  0.21 

Exploration  nominal  —  Exploration  off-nominal 

Z  =  -1.18, 
p  =  0.24 

Z  =  -2.58, 

p  =  0.01 

Z  =  -1.78, 
p  =  0.074 

Z  =  -0.62, 
p  =  0.53 

Non-optimal  satisficing 

Descriptive  statistics  for  non-optimal  satisficing  (cost  surplus) 


Visualization 

Nominal 

Off-nominal  1 

Group 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Elevation 

Contours 

0.00  ±  0.00 

0.05  ±  0.15 

0.08  ±  0.17 

0.20  ±  0.39 

6.18  ±  9.02 

1.01  ±  1.26 

LOEC 

0.05  ±  0.12 

0.84  ±  2.04 

0.17  ±  0.24 

0.07  ±  0.13 

3.86  ±  6.04 

1.05  ±  1.44 

Both 

visualizations 

0.00  ±  0.00 

1.09  ±  2.88 

0.04  ±  0.14 

0.19  ±  0.40 

4.18  ±  8.03 

1.42  ±  2.18 

Total 

0.02  ±  0.07 

0.66  ±  2.02 

0.10  ±  0.19 

0.16  ±  0.33 

4.80  ±  7.66 

1.15  ±  1.60 
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Descriptive  statistics  for  non-optimal  satisficing  (time  surplus) 


Visualization 

Group 

Nominal 

Off-Nominal 

Distance 

Time 

Metabolic 

Exploration 

Time 

Exploration 

Elevation 

Contours 

27.14%  ± 
10.87 

21.68%  ± 
19.89 

32.53%  ± 
28.83 

42.92%  ± 
27.16 

36.73%  ± 
22.10 

54.69%  ± 
28.39 

LOEC 

23.21%  ± 
21.01 

18.62%  ± 
22.78 

39.09%  ± 
23.19 

32.80%  ± 
20.52 

30.38%  ± 
31.45 

38.42%  ± 
28.44 

Both 

visualizations 

33.87%  ± 
25.99 

20.37%  ± 
22.48 

31.65%  ± 
28.33 

29.36%  ± 
23.49 

29.05%  ± 
28.56 

34.78%  ± 
30.92 

Total 

28.04%  ± 
19.93 

20.27%  ± 
21.07 

34.37%  ± 
26.35 

35.26%  ± 
23.99 

32.19%  ± 
26.86 

42.99%  ± 
29.71 

Repeated  analysis  of  variance  for  Phase  1,  time  surplus 


Factor 

F-value 

p -value 

Cost  function 

F(2.21,  68.56)  =  3.11 

p  =  0.046 

Visualizations 

F(2,  31)  =  0.15 

p  =  0.86 

Cost  function  x  Visualization 

F(4.42,  68.56)  =  0.65 

p  =  0.65 

Simple  contrast  of  time  surplus,  Phase  1 


NOMINAL 

Distance 

Metabolic 

Exploration 

Time 

F(l,31)  =  2.27,  p  =  0.14 

F(l,31)  =  12.76,  p  =  0.001 

F(l,31)  =  6.24,  p  =  0.018 

Point-wise  comparisons  for  Exploration  function  (time  surplus),  least  significant  difference 
test 


NOMINAL 

LOEC 

Elevation  Contours  & 
LOEC 

Elevation  Contours 

p  =  0.32 

p  =  0.19 

LOEC 

p  =  0.74 

Kruskal- Wallis  test  for  non-optimal  satisficing  (cost  surplus),  Phase  2 


Cost  Function 

Scenario 

Kruskal-Wallis  test 

p-value 

Time 

Off-nominal 

X2(2,  33)  =  0.77 

p  =  0.68 

Exploration 

Off-nominal 

X2(2,  33)  =  0.25 

p  =  0.88 
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Wilcoxon  Sign  test  results  of  pair-wise  comparisons  for  cost  surplus  between  cost  functions, 
Phase  2 


Cost  Function  Comparisons 

Wilcoxon  Sign  test 

Time  off-nominal  —  Exploration  off-nominal 

Z  =  -3.36,  p  =  0.001 

Time  nominal  —  Time  off-nominal 

Z  =  -2.14,  p  =  0.032 

Exploration  nominal  —  Exploration  off-nominal 

Z  = -2.54,  p  =  0.011 

Repeated  analysis  of  variance  (time  surplus)  for  Phase  2 


Factor 

F-value 

p -value 

Cost  function 

F(l,  31)  =  8.24 

p  =  0.007 

Visualizations 

F(2,31)  =  2.07 

p  =  0.14 

Scenario 

F(1 ,31)  =  4.96 

p  =  0.033 

Cost  function  x  Visualization 

F(2,31)  =  0.69 

p  =  0.51 

Scenario  x  Visualization 

F(2,31)  =  0.20 

p  =  0.82 

Function  x  Scenario 

F(l,31)  =  0.25 

p  =  0.62 

Function  x  Scenario  x  Visualization 

F(2,31)  =  0.012 

p  =  0.99 

•  Correlations  time  surplus  versus  true  time  (off-nominal  condition): 

o  Time  function,  Pearson  correlation  test  =  -0.66,  p  <  0.0001. 
o  Exploration  function,  Pearson  correlation  test  =  -0.73,  p  <  0.0001 


Other  Analyses 

Kolmogorov-Smirnov  normality  tests 


Dependent  variable 

K-S  test,  Z 

p-value 

Path  cost  errors 

Z  =  4.34 

p  <  0.0001 

Time  penalty 

Z  =  1.99 

p  =  0.001 

Total  time 

Z  =  1.19 

p  =  0.12 

True  time 

Z  =  1.41 

p  =  0.037 

Spearman’s  rho  correlations  between  path  cost  errors  and  time  dependent  variable 


Time  dependent  variable 

Correlation  coefficient 

p-value 

Time  penalty 

0.082 

p  =  0.25 

Total  time 

0.090 

p  =  0.20 

True  time 

0.065 

p  =  0.36 
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Elevation  contours  Test  trial  LOEC  Test  trial 


Elevation  contours  Practice  tri  a  I _ LOEC  Practice  trial 


40- 

30- 


3 


02460246 


Both  visualizations  Test  trial 


Both  visualizations  Practice  trial 


0  2  4  6 


Tab  clicks 


Tab  clicks 


Tab  clicks 


Histograms  of  total  instances  cost  function  tab  was  clicked.  Columns,  visualization  types; 
top  row,  nominal  test  trials;  bottom  row,  practice  trials 


Elevation  contours  LOEC 


Both  visualizations 


0  2  4  6 


Tab  clicks 


Tab  clicks 


Tab  clicks 


Histograms  of  total  instances  cost  function  tab  was  clicked,  off-nominal  test  trials  only. 

Columns,  visualization  types. 
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Cognitive  Strategies 


Descriptive  statistics  of  path  cost  errors  and  Mann-Whitney  test  comparisons  across  strategy 
types  for  path  cost  errors 


Cost  Function 

Manual  sensitivity 
analysis 

“Whole-path” 
sensitivity  analysis 

Mann-Whitney  test 
&  p-value 

Distance 

1.03  ±  0.03 

1.02  ±  0.03 

Z  =  -0.69,  p  =  0.49 

Time 

1.60  ±  0.14 

1.61  ±  0.15 

Z  =  -0.37,  p  =  0.72 

Metabolic 

1.07  ±  0.02 

1.08  ±  0.04 

Z  =  -0.75,  p  =  0.45 

Exploration 

1.06  ±  0.02 

1.06  ±  0.02 

Z  =  -0.28,  p  =  0.78 

Time  (off-nominal) 

1.51  ±  0.20 

1.59  ±  0.20 

Z  =  -1.14,  p  =  0.26 

Exploration  (off-nominal) 

1.10  ±  0.03 

1.12  ±  0.05 

Z  =  -0.73,  p  =  0.47 

Descriptive  statistics  of  distance  errors 


Cost  Function 

Path  error  based  on 
distance 

Distance 

1.03  ±  0.04 

Time 

1.12  ±  0.07 

Metabolic 

1.13  ±  0.04 

Exploration 

1.06  ±  0.04 

Time  (off-nominal) 

1.08  ±  0.04 

Exploration  (off-nominal) 

1.07  ±  0.05 

Repeated  analysis  of  variance  for  nominal  conditions,  distance  errors 


Factor 

F-value 

p-value 

Cost  function 

F(l,99)  =  34.48 

p  <  0.0001 

•  Visualization  was  not  used  as  a  test  condition  to  analyze  shortest  path  heuristic 
because  previous  experimental  results  indicate  that  there  is  no  main  effect  due  to 
visualization.  Furthermore,  including  it  as  a  condition  did  not  reveal  significant 
differences  among  visualization  groups. 

Simple  contrast  of  distance  errors,  nominal  conditions 


NOMINAL 

Time 

Metabolic 

Exploration 

Distance 

F(1 , 33)  =  48.3,  p  <  0.0001 

F(1 , 33)  =  116.0,  p<  0.0001 

F(l,33)  =  7.20,  p  =  0.011 

Time 

F(l,33)  =  0.72,  p  =  0.40 

F(l,33)  =  19.5,  p  <  0.0001 

Metabolic 

F(l,33)  =  53.0,  p  <  0.0001 
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Repeated  analysis  of  variance  for  off-nominal  conditions,  distance  errors 


Factor 

F-value 

p -value 

Cost  function 

F(l,33)  =  19.46 

p  <  0.0001 

Scenario 

F(l,33)  =  3.05 

p  =  0.09 

Cost  function  x  Scenario 

F(l,33)  =  9.61 

p  =  0.004 

Point-wise  comparisons  across  scenarios  within  cost  function  (distance  errors),  least 
significant  difference  test 


Nominal  vs.  Off-Nominal 
p-value 

Time 

p  =  0.003 

Exploration 

p  =  0.36 
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Appendix  C:  Meta-Analysis  Assessing 
Sensitivity  Analysis 


A  meta-analysis  was  conducted  in  order  to  assess  the  effect  of  sensitivity  analysis  across  both 
experiments.  This  analysis  would  include  all  participants,  61  in  total,  and  include  all  test  trials  (a  total 
of  8).  Thus,  participants  were  ranked  in  the  following  manner: 

•  Every  participant  was  assigned  a  sensitivity  analysis  type:  0  if  they  chose  whole-path 
sensitivity  analysis,  and  1  if  they  chose  path  modifications. 

•  For  every  participant  and  for  each  function,  a  performance  score  was  assigned.  Performance 
scores  ranged  from  1  to  3.  A  score  of  1  indicated  that  the  participant  had  one  of  the 
smallest  path  cost  errors  for  that  trial.  Similarly,  3  indicated  poor  performance.  Score  was 
assigned  based  on  the  average  and  standard  deviations  for  that  trial.  A  score  of  2  was 
awarded  to  participants  whose  path  cost  errors  were  ±  1  standard  deviation  from  the  mean 
error  of  that  trial.  Above  and  below  one  SD,  these  participants  were  given  scores  of  1  and  3, 
respectively. 

•  For  every  participant  and  for  each  function,  a  ranking  of  modification  time  was  given.  Since 
the  percent  modifying  time  ranged  from  0  to  85%,  9  bins  were  created,  one  for  every  10 
percent  increment. 
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